Lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integration
Abstract Accurately identifying the stages of lung adenocarcinoma is essential for selecting the most appropriate treatment plans. Nonetheless, this task is complicated due to challenges such as integrating diverse data, similarities among subtypes, and the need to capture contextual features, makin...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-08-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-13818-2 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849226530654257152 |
|---|---|
| author | Changmiao Wang Lijian Liu Chenchen Fan Yongquan Zhang Zhijun Mai Li Li Zhou Liu Yuan Tian Jiahang Hu Ahmed Elazab |
| author_facet | Changmiao Wang Lijian Liu Chenchen Fan Yongquan Zhang Zhijun Mai Li Li Zhou Liu Yuan Tian Jiahang Hu Ahmed Elazab |
| author_sort | Changmiao Wang |
| collection | DOAJ |
| description | Abstract Accurately identifying the stages of lung adenocarcinoma is essential for selecting the most appropriate treatment plans. Nonetheless, this task is complicated due to challenges such as integrating diverse data, similarities among subtypes, and the need to capture contextual features, making precise differentiation difficult. We address these challenges and propose a multimodal deep neural network that integrates computed tomography (CT) images, annotated lesion bounding boxes, and electronic health records. Our model first combines bounding boxes with precise lesion location data and CT scans, generating a richer semantic representation through feature extraction from regions of interest to enhance localization accuracy using a vision transformer module. Beyond imaging data, the model also incorporates clinical information encoded using a fully connected encoder. Features extracted from both CT and clinical data are optimized for cosine similarity using a contrastive language-image pre-training module, ensuring they are cohesively integrated. In addition, we introduce an attention-based feature fusion module that harmonizes these features into a unified representation to fuse features from different modalities further. This integrated feature set is then fed into a classifier that effectively distinguishes among the three types of adenocarcinomas. Finally, we employ focal loss to mitigate the effects of unbalanced classes and contrastive learning loss to enhance feature representation and improve the model’s performance. Our experiments on public and proprietary datasets demonstrate the efficiency of our model, achieving a superior validation accuracy of 81.42% and an area under the curve of 0.9120. These results significantly outperform recent multimodal classification approaches. The code is available at https://github.com/fancccc/LungCancerDC . |
| format | Article |
| id | doaj-art-d34eff4ed89e4524b1ee8246daf22a55 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-08-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-d34eff4ed89e4524b1ee8246daf22a552025-08-24T11:17:13ZengNature PortfolioScientific Reports2045-23222025-08-0115111210.1038/s41598-025-13818-2Lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integrationChangmiao Wang0Lijian Liu1Chenchen Fan2Yongquan Zhang3Zhijun Mai4Li Li5Zhou Liu6Yuan Tian7Jiahang Hu8Ahmed Elazab9Shenzhen Research Institute of Big DataNational Cancer Center, Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Cancer, Cancer Hospital and Shenzhen HospitalZhejiang University of Finance and EconomicsZhejiang University of Finance and EconomicsNational Cancer Center, Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Cancer, Cancer Hospital and Shenzhen HospitalNational Cancer Center, Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Cancer, Cancer Hospital and Shenzhen HospitalNational Cancer Center, Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Cancer, Cancer Hospital and Shenzhen HospitalThe Second Affiliated Hospital, School of Medicine, The Chinese University of Hong KongHongqi Hospital Affiliated to Mudanjiang Medical UniversitySchool of Biomedical Engineering, Shenzhen UniversityAbstract Accurately identifying the stages of lung adenocarcinoma is essential for selecting the most appropriate treatment plans. Nonetheless, this task is complicated due to challenges such as integrating diverse data, similarities among subtypes, and the need to capture contextual features, making precise differentiation difficult. We address these challenges and propose a multimodal deep neural network that integrates computed tomography (CT) images, annotated lesion bounding boxes, and electronic health records. Our model first combines bounding boxes with precise lesion location data and CT scans, generating a richer semantic representation through feature extraction from regions of interest to enhance localization accuracy using a vision transformer module. Beyond imaging data, the model also incorporates clinical information encoded using a fully connected encoder. Features extracted from both CT and clinical data are optimized for cosine similarity using a contrastive language-image pre-training module, ensuring they are cohesively integrated. In addition, we introduce an attention-based feature fusion module that harmonizes these features into a unified representation to fuse features from different modalities further. This integrated feature set is then fed into a classifier that effectively distinguishes among the three types of adenocarcinomas. Finally, we employ focal loss to mitigate the effects of unbalanced classes and contrastive learning loss to enhance feature representation and improve the model’s performance. Our experiments on public and proprietary datasets demonstrate the efficiency of our model, achieving a superior validation accuracy of 81.42% and an area under the curve of 0.9120. These results significantly outperform recent multimodal classification approaches. The code is available at https://github.com/fancccc/LungCancerDC .https://doi.org/10.1038/s41598-025-13818-2Lung AdenocarcinomaClinical InformationMultimodal Learning |
| spellingShingle | Changmiao Wang Lijian Liu Chenchen Fan Yongquan Zhang Zhijun Mai Li Li Zhou Liu Yuan Tian Jiahang Hu Ahmed Elazab Lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integration Scientific Reports Lung Adenocarcinoma Clinical Information Multimodal Learning |
| title | Lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integration |
| title_full | Lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integration |
| title_fullStr | Lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integration |
| title_full_unstemmed | Lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integration |
| title_short | Lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integration |
| title_sort | lung adenocarcinoma subtype classification based on contrastive learning model with multimodal integration |
| topic | Lung Adenocarcinoma Clinical Information Multimodal Learning |
| url | https://doi.org/10.1038/s41598-025-13818-2 |
| work_keys_str_mv | AT changmiaowang lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT lijianliu lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT chenchenfan lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT yongquanzhang lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT zhijunmai lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT lili lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT zhouliu lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT yuantian lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT jiahanghu lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration AT ahmedelazab lungadenocarcinomasubtypeclassificationbasedoncontrastivelearningmodelwithmultimodalintegration |