Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images
Objectives: Early skin cancer detection in primary care settings is crucial for prognosis, yet clinicians often lack relevant training. Machine learning (ML) methods may offer a potential solution for this dilemma. This study aimed to develop a neural network for the binary classification of skin le...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SAGE Publishing
2025-06-01
|
| Series: | Cancer Informatics |
| Online Access: | https://doi.org/10.1177/11769351251349891 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849682854156435456 |
|---|---|
| author | Jiawen Deng Eddie Guo Heather Jianbo Zhao Kaden Venugopal Myron Moskalyk |
| author_facet | Jiawen Deng Eddie Guo Heather Jianbo Zhao Kaden Venugopal Myron Moskalyk |
| author_sort | Jiawen Deng |
| collection | DOAJ |
| description | Objectives: Early skin cancer detection in primary care settings is crucial for prognosis, yet clinicians often lack relevant training. Machine learning (ML) methods may offer a potential solution for this dilemma. This study aimed to develop a neural network for the binary classification of skin lesions into malignant and benign categories using smartphone images and clinical data via a multimodal and transfer learning-based approach. Methods: We used the PAD-UFES-20 dataset, which included 2298 sets of lesion images. Three neural network models were developed: (1) a clinical data-based network, (2) an image-based network using a pre-trained DenseNet-121 and (3) a multimodal network combining clinical and image data. Models were tuned using Bayesian Optimisation HyperBand across 5-fold cross-validation. Model performance was evaluated using AUC-ROC, average precision, Brier score, calibration curve metrics, Matthews correlation coefficient (MCC), sensitivity and specificity. Model explainability was explored using permutation importance and Grad-CAM. Results: During cross-validation, the multimodal network achieved an AUC-ROC of 0.91 (95% confidence interval [CI] 0.88-0.93) and a Brier score of 0.15 (95% CI 0.11-0.19). During internal validation, it retained an AUC-ROC of 0.91 and a Brier score of 0.12. The multimodal network outperformed the unimodal models on threshold-independent metrics and at MCC-optimised threshold, but it had similar classification performance as the image-only model at high-sensitivity thresholds. Analysis of permutation importance showed that key clinical features influential for the clinical data-based network included bleeding, lesion elevation, patient age and recent lesion growth. Grad-CAM visualisations showed that the image-based network focused on lesioned regions during classification rather than background artefacts. Conclusions: A transfer learning-based, multimodal neural network can accurately identify malignant skin lesions from smartphone images and clinical data. External validation with larger, more diverse datasets is needed to assess the model’s generalisability and support clinical adoption. |
| format | Article |
| id | doaj-art-1ff6c8b7fe4447879d7202918e89dd8c |
| institution | DOAJ |
| issn | 1176-9351 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | SAGE Publishing |
| record_format | Article |
| series | Cancer Informatics |
| spelling | doaj-art-1ff6c8b7fe4447879d7202918e89dd8c2025-08-20T03:24:03ZengSAGE PublishingCancer Informatics1176-93512025-06-012410.1177/11769351251349891Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone ImagesJiawen Deng0Eddie Guo1Heather Jianbo Zhao2Kaden Venugopal3Myron Moskalyk4Temerty Faculty of Medicine, University of Toronto, Toronto, ON, CanadaCumming School of Medicine, University of Calgary, Calgary, AB, CanadaTemerty Faculty of Medicine, University of Toronto, Toronto, ON, CanadaFaculty of Health Sciences, University of Ottawa, Ottawa, ON, CanadaBiostatistics Division, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, CanadaObjectives: Early skin cancer detection in primary care settings is crucial for prognosis, yet clinicians often lack relevant training. Machine learning (ML) methods may offer a potential solution for this dilemma. This study aimed to develop a neural network for the binary classification of skin lesions into malignant and benign categories using smartphone images and clinical data via a multimodal and transfer learning-based approach. Methods: We used the PAD-UFES-20 dataset, which included 2298 sets of lesion images. Three neural network models were developed: (1) a clinical data-based network, (2) an image-based network using a pre-trained DenseNet-121 and (3) a multimodal network combining clinical and image data. Models were tuned using Bayesian Optimisation HyperBand across 5-fold cross-validation. Model performance was evaluated using AUC-ROC, average precision, Brier score, calibration curve metrics, Matthews correlation coefficient (MCC), sensitivity and specificity. Model explainability was explored using permutation importance and Grad-CAM. Results: During cross-validation, the multimodal network achieved an AUC-ROC of 0.91 (95% confidence interval [CI] 0.88-0.93) and a Brier score of 0.15 (95% CI 0.11-0.19). During internal validation, it retained an AUC-ROC of 0.91 and a Brier score of 0.12. The multimodal network outperformed the unimodal models on threshold-independent metrics and at MCC-optimised threshold, but it had similar classification performance as the image-only model at high-sensitivity thresholds. Analysis of permutation importance showed that key clinical features influential for the clinical data-based network included bleeding, lesion elevation, patient age and recent lesion growth. Grad-CAM visualisations showed that the image-based network focused on lesioned regions during classification rather than background artefacts. Conclusions: A transfer learning-based, multimodal neural network can accurately identify malignant skin lesions from smartphone images and clinical data. External validation with larger, more diverse datasets is needed to assess the model’s generalisability and support clinical adoption.https://doi.org/10.1177/11769351251349891 |
| spellingShingle | Jiawen Deng Eddie Guo Heather Jianbo Zhao Kaden Venugopal Myron Moskalyk Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images Cancer Informatics |
| title | Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images |
| title_full | Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images |
| title_fullStr | Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images |
| title_full_unstemmed | Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images |
| title_short | Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images |
| title_sort | development of a transfer learning based multimodal neural network for identifying malignant dermatological lesions from smartphone images |
| url | https://doi.org/10.1177/11769351251349891 |
| work_keys_str_mv | AT jiawendeng developmentofatransferlearningbasedmultimodalneuralnetworkforidentifyingmalignantdermatologicallesionsfromsmartphoneimages AT eddieguo developmentofatransferlearningbasedmultimodalneuralnetworkforidentifyingmalignantdermatologicallesionsfromsmartphoneimages AT heatherjianbozhao developmentofatransferlearningbasedmultimodalneuralnetworkforidentifyingmalignantdermatologicallesionsfromsmartphoneimages AT kadenvenugopal developmentofatransferlearningbasedmultimodalneuralnetworkforidentifyingmalignantdermatologicallesionsfromsmartphoneimages AT myronmoskalyk developmentofatransferlearningbasedmultimodalneuralnetworkforidentifyingmalignantdermatologicallesionsfromsmartphoneimages |