Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion Recognition
The rising prevalence of skin lesions places a heavy burden on global health resources and necessitates an early and precise diagnosis for successful treatment. The diagnostic potential of recent multi-modal skin lesion detection algorithms is limited because they ignore dynamic interactions and inf...
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-03-01
|
| Series: | Bioengineering |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2306-5354/12/3/282 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850089768726036480 |
|---|---|
| author | Yingzhe Yu Huiqiong Jia Li Zhang Suling Xu Xiaoxia Zhu Jiucun Wang Fangfang Wang Lianyi Han Haoqiang Jiang Qiongyan Zhou Chao Xin |
| author_facet | Yingzhe Yu Huiqiong Jia Li Zhang Suling Xu Xiaoxia Zhu Jiucun Wang Fangfang Wang Lianyi Han Haoqiang Jiang Qiongyan Zhou Chao Xin |
| author_sort | Yingzhe Yu |
| collection | DOAJ |
| description | The rising prevalence of skin lesions places a heavy burden on global health resources and necessitates an early and precise diagnosis for successful treatment. The diagnostic potential of recent multi-modal skin lesion detection algorithms is limited because they ignore dynamic interactions and information sharing across modalities at various feature scales. To address this, we propose a deep learning framework, Multi-Modal Skin-Imaging-based Information-Switching Network (MDSIS-Net), for end-to-end skin lesion recognition. MDSIS-Net extracts intra-modality features using transfer learning in a multi-scale fully shared convolutional neural network and introduces an innovative information-switching module. A cross-attention mechanism dynamically calibrates and integrates features across modalities to improve inter-modality associations and feature representation in this module. MDSIS-Net is tested on clinical disfiguring dermatosis data and the public Derm7pt melanoma dataset. A Visually Intelligent System for Image Analysis (VISIA) captures five modalities: spots, red marks, ultraviolet (UV) spots, porphyrins, and brown spots for disfiguring dermatosis. The model performs better than existing approaches with an mAP of 0.967, accuracy of 0.960, precision of 0.935, recall of 0.960, and f1-score of 0.947. Using clinical and dermoscopic pictures from the Derm7pt dataset, MDSIS-Net outperforms current benchmarks for melanoma, with an mAP of 0.877, accuracy of 0.907, precision of 0.911, recall of 0.815, and f1-score of 0.851. The model’s interpretability is proven by Grad-CAM heatmaps correlating with clinical diagnostic focus areas. In conclusion, our deep multi-modal information-switching model enhances skin lesion identification by capturing relationship features and fine-grained details across multi-modal images, improving both accuracy and interpretability. This work advances clinical decision making and lays a foundation for future developments in skin lesion diagnosis and treatment. |
| format | Article |
| id | doaj-art-65408b057c4b429da0c41f9ca5e8f6e9 |
| institution | DOAJ |
| issn | 2306-5354 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Bioengineering |
| spelling | doaj-art-65408b057c4b429da0c41f9ca5e8f6e92025-08-20T02:42:41ZengMDPI AGBioengineering2306-53542025-03-0112328210.3390/bioengineering12030282Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion RecognitionYingzhe Yu0Huiqiong Jia1Li Zhang2Suling Xu3Xiaoxia Zhu4Jiucun Wang5Fangfang Wang6Lianyi Han7Haoqiang Jiang8Qiongyan Zhou9Chao Xin10The First Affiliated Hospital of Ningbo University, Ningbo 315211, ChinaDepartment of Laboratory Medicine, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou 310000, ChinaDepartment of Dermatology, The First Hospital of China Medical University, Shenyang 110001, ChinaThe First Affiliated Hospital of Ningbo University, Ningbo 315211, ChinaThe First Affiliated Hospital of Ningbo University, Ningbo 315211, ChinaState Key Laboratory of Genetic Engineering, Collaborative Innovation Center for Genetics and Development, School of Life Sciences and Human Phenome Institute, Fudan University, Shanghai 200433, ChinaThe First Affiliated Hospital of Ningbo University, Ningbo 315211, ChinaGreater Bay Area Institute of Precision Medicine (Guangzhou), School of Life Sciences, Fudan University, Shanghai 315211, ChinaGreater Bay Area Institute of Precision Medicine (Guangzhou), School of Life Sciences, Fudan University, Shanghai 315211, ChinaThe First Affiliated Hospital of Ningbo University, Ningbo 315211, ChinaThe First Affiliated Hospital of Ningbo University, Ningbo 315211, ChinaThe rising prevalence of skin lesions places a heavy burden on global health resources and necessitates an early and precise diagnosis for successful treatment. The diagnostic potential of recent multi-modal skin lesion detection algorithms is limited because they ignore dynamic interactions and information sharing across modalities at various feature scales. To address this, we propose a deep learning framework, Multi-Modal Skin-Imaging-based Information-Switching Network (MDSIS-Net), for end-to-end skin lesion recognition. MDSIS-Net extracts intra-modality features using transfer learning in a multi-scale fully shared convolutional neural network and introduces an innovative information-switching module. A cross-attention mechanism dynamically calibrates and integrates features across modalities to improve inter-modality associations and feature representation in this module. MDSIS-Net is tested on clinical disfiguring dermatosis data and the public Derm7pt melanoma dataset. A Visually Intelligent System for Image Analysis (VISIA) captures five modalities: spots, red marks, ultraviolet (UV) spots, porphyrins, and brown spots for disfiguring dermatosis. The model performs better than existing approaches with an mAP of 0.967, accuracy of 0.960, precision of 0.935, recall of 0.960, and f1-score of 0.947. Using clinical and dermoscopic pictures from the Derm7pt dataset, MDSIS-Net outperforms current benchmarks for melanoma, with an mAP of 0.877, accuracy of 0.907, precision of 0.911, recall of 0.815, and f1-score of 0.851. The model’s interpretability is proven by Grad-CAM heatmaps correlating with clinical diagnostic focus areas. In conclusion, our deep multi-modal information-switching model enhances skin lesion identification by capturing relationship features and fine-grained details across multi-modal images, improving both accuracy and interpretability. This work advances clinical decision making and lays a foundation for future developments in skin lesion diagnosis and treatment.https://www.mdpi.com/2306-5354/12/3/282skin lesiondeep multi-modal networkrecognitioninformation switching |
| spellingShingle | Yingzhe Yu Huiqiong Jia Li Zhang Suling Xu Xiaoxia Zhu Jiucun Wang Fangfang Wang Lianyi Han Haoqiang Jiang Qiongyan Zhou Chao Xin Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion Recognition Bioengineering skin lesion deep multi-modal network recognition information switching |
| title | Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion Recognition |
| title_full | Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion Recognition |
| title_fullStr | Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion Recognition |
| title_full_unstemmed | Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion Recognition |
| title_short | Deep Multi-Modal Skin-Imaging-Based Information-Switching Network for Skin Lesion Recognition |
| title_sort | deep multi modal skin imaging based information switching network for skin lesion recognition |
| topic | skin lesion deep multi-modal network recognition information switching |
| url | https://www.mdpi.com/2306-5354/12/3/282 |
| work_keys_str_mv | AT yingzheyu deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT huiqiongjia deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT lizhang deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT sulingxu deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT xiaoxiazhu deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT jiucunwang deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT fangfangwang deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT lianyihan deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT haoqiangjiang deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT qiongyanzhou deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition AT chaoxin deepmultimodalskinimagingbasedinformationswitchingnetworkforskinlesionrecognition |