Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic Images
Background: Multimodal large language models (LLMs) have shown potential in various health-related fields. However, many healthcare studies have raised concerns about the reliability and biases of LLMs in healthcare applications. Methods: To explore the practical application of multimodal LLMs in sk...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Association for the Advancement of Science (AAAS)
2025-01-01
|
| Series: | Health Data Science |
| Online Access: | https://spj.science.org/doi/10.34133/hds.0256 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849739640876040192 |
|---|---|
| author | Zhiyu Wan Yuhang Guo Shunxing Bao Qian Wang Bradley A. Malin |
| author_facet | Zhiyu Wan Yuhang Guo Shunxing Bao Qian Wang Bradley A. Malin |
| author_sort | Zhiyu Wan |
| collection | DOAJ |
| description | Background: Multimodal large language models (LLMs) have shown potential in various health-related fields. However, many healthcare studies have raised concerns about the reliability and biases of LLMs in healthcare applications. Methods: To explore the practical application of multimodal LLMs in skin disease identification, and to evaluate sex and age biases, we tested the performance of 2 popular multimodal LLMs, ChatGPT-4 and LLaVA-1.6, across diverse sex and age groups using a subset of a large dermatoscopic dataset containing around 10,000 images and 3 skin diseases (melanoma, melanocytic nevi, and benign keratosis-like lesions). Results: In comparison to 3 deep learning models (VGG16, ResNet50, and Model Derm) based on convolutional neural network (CNN), one vision transformer model (Swin-B), we found that ChatGPT-4 and LLaVA-1.6 demonstrated overall accuracies that were 3% and 23% higher (and F1-scores that were 4% and 34% higher), respectively, than the best performing CNN-based baseline while maintaining accuracies that were 38% and 26% lower (and F1-scores that were 38% and 19% lower), respectively, than Swin-B. Meanwhile, ChatGPT-4 is generally unbiased in identifying these skin diseases across sex and age groups, while LLaVA-1.6 is generally unbiased across age groups, in contrast to Swin-B, which is biased in identifying melanocytic nevi. Conclusions: This study suggests the usefulness and fairness of LLMs in dermatological applications, aiding physicians and practitioners with diagnostic recommendations and patient screening. To further verify and evaluate the reliability and fairness of LLMs in healthcare, experiments using larger and more diverse datasets need to be performed in the future. |
| format | Article |
| id | doaj-art-e9e6f30fa59840ad9a06a6506d8a0760 |
| institution | DOAJ |
| issn | 2765-8783 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | American Association for the Advancement of Science (AAAS) |
| record_format | Article |
| series | Health Data Science |
| spelling | doaj-art-e9e6f30fa59840ad9a06a6506d8a07602025-08-20T03:06:13ZengAmerican Association for the Advancement of Science (AAAS)Health Data Science2765-87832025-01-01510.34133/hds.0256Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic ImagesZhiyu Wan0Yuhang Guo1Shunxing Bao2Qian Wang3Bradley A. Malin4Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA.School of Biomedical Engineering, ShanghaiTech University, Shanghai, China.Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA.School of Biomedical Engineering, ShanghaiTech University, Shanghai, China.Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA.Background: Multimodal large language models (LLMs) have shown potential in various health-related fields. However, many healthcare studies have raised concerns about the reliability and biases of LLMs in healthcare applications. Methods: To explore the practical application of multimodal LLMs in skin disease identification, and to evaluate sex and age biases, we tested the performance of 2 popular multimodal LLMs, ChatGPT-4 and LLaVA-1.6, across diverse sex and age groups using a subset of a large dermatoscopic dataset containing around 10,000 images and 3 skin diseases (melanoma, melanocytic nevi, and benign keratosis-like lesions). Results: In comparison to 3 deep learning models (VGG16, ResNet50, and Model Derm) based on convolutional neural network (CNN), one vision transformer model (Swin-B), we found that ChatGPT-4 and LLaVA-1.6 demonstrated overall accuracies that were 3% and 23% higher (and F1-scores that were 4% and 34% higher), respectively, than the best performing CNN-based baseline while maintaining accuracies that were 38% and 26% lower (and F1-scores that were 38% and 19% lower), respectively, than Swin-B. Meanwhile, ChatGPT-4 is generally unbiased in identifying these skin diseases across sex and age groups, while LLaVA-1.6 is generally unbiased across age groups, in contrast to Swin-B, which is biased in identifying melanocytic nevi. Conclusions: This study suggests the usefulness and fairness of LLMs in dermatological applications, aiding physicians and practitioners with diagnostic recommendations and patient screening. To further verify and evaluate the reliability and fairness of LLMs in healthcare, experiments using larger and more diverse datasets need to be performed in the future.https://spj.science.org/doi/10.34133/hds.0256 |
| spellingShingle | Zhiyu Wan Yuhang Guo Shunxing Bao Qian Wang Bradley A. Malin Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic Images Health Data Science |
| title | Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic Images |
| title_full | Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic Images |
| title_fullStr | Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic Images |
| title_full_unstemmed | Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic Images |
| title_short | Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic Images |
| title_sort | evaluating sex and age biases in multimodal large language models for skin disease identification from dermatoscopic images |
| url | https://spj.science.org/doi/10.34133/hds.0256 |
| work_keys_str_mv | AT zhiyuwan evaluatingsexandagebiasesinmultimodallargelanguagemodelsforskindiseaseidentificationfromdermatoscopicimages AT yuhangguo evaluatingsexandagebiasesinmultimodallargelanguagemodelsforskindiseaseidentificationfromdermatoscopicimages AT shunxingbao evaluatingsexandagebiasesinmultimodallargelanguagemodelsforskindiseaseidentificationfromdermatoscopicimages AT qianwang evaluatingsexandagebiasesinmultimodallargelanguagemodelsforskindiseaseidentificationfromdermatoscopicimages AT bradleyamalin evaluatingsexandagebiasesinmultimodallargelanguagemodelsforskindiseaseidentificationfromdermatoscopicimages |