A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis
Abstract Radiology images are one of the most commonly used in daily clinical diagnosis. Typically, clinical diagnosis using radiology images involves disease reporting and classification, where the former is a multimodal task whereby textual reports are generated to describe clinical findings in im...
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-02-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-024-01339-7 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823861571061809152 |
|---|---|
| author | Fenglin Liu Zheng Li Qingyu Yin Jinfa Huang Jiebo Luo Anshul Thakur Kim Branson Patrick Schwab Bing Yin Xian Wu Yefeng Zheng David A. Clifton |
| author_facet | Fenglin Liu Zheng Li Qingyu Yin Jinfa Huang Jiebo Luo Anshul Thakur Kim Branson Patrick Schwab Bing Yin Xian Wu Yefeng Zheng David A. Clifton |
| author_sort | Fenglin Liu |
| collection | DOAJ |
| description | Abstract Radiology images are one of the most commonly used in daily clinical diagnosis. Typically, clinical diagnosis using radiology images involves disease reporting and classification, where the former is a multimodal task whereby textual reports are generated to describe clinical findings in images, as are common in various domains, e.g., chest X-ray or computed tomography. Existing approaches are mainly supervised, the quality of which heavily depends on the volume and quality of available labeled data. However, for rarer or more novel diseases, enrolling patients to collect data is both time-consuming and expensive. For non-English languages, sufficient quantities of labeled data are typically not available. We propose the Multimodal Multidomain Multilingual Foundation Model. It is useful for rare diseases and non-English languages, where the labeled data are frequently much more scarce, and may even be absent. Our approach achieves encouraging performances on nine datasets, including 2 infectious and 14 non-infectious diseases. |
| format | Article |
| id | doaj-art-7dd82fd56c034dcb9a938afd819d65ea |
| institution | Kabale University |
| issn | 2398-6352 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj-art-7dd82fd56c034dcb9a938afd819d65ea2025-02-09T12:55:37ZengNature Portfolionpj Digital Medicine2398-63522025-02-018111210.1038/s41746-024-01339-7A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosisFenglin Liu0Zheng Li1Qingyu Yin2Jinfa Huang3Jiebo Luo4Anshul Thakur5Kim Branson6Patrick Schwab7Bing Yin8Xian Wu9Yefeng Zheng10David A. Clifton11Institute of Biomedical Engineering, Department of Engineering Science, University of OxfordAmazonAmazonDepartment of Computer Science, University of RochesterDepartment of Computer Science, University of RochesterInstitute of Biomedical Engineering, Department of Engineering Science, University of OxfordGlaxoSmithKlineGlaxoSmithKlineAmazonJarvis Research Center, Tencent YouTu LabMedical Artificial Intelligence Laboratory, Westlake UniversityInstitute of Biomedical Engineering, Department of Engineering Science, University of OxfordAbstract Radiology images are one of the most commonly used in daily clinical diagnosis. Typically, clinical diagnosis using radiology images involves disease reporting and classification, where the former is a multimodal task whereby textual reports are generated to describe clinical findings in images, as are common in various domains, e.g., chest X-ray or computed tomography. Existing approaches are mainly supervised, the quality of which heavily depends on the volume and quality of available labeled data. However, for rarer or more novel diseases, enrolling patients to collect data is both time-consuming and expensive. For non-English languages, sufficient quantities of labeled data are typically not available. We propose the Multimodal Multidomain Multilingual Foundation Model. It is useful for rare diseases and non-English languages, where the labeled data are frequently much more scarce, and may even be absent. Our approach achieves encouraging performances on nine datasets, including 2 infectious and 14 non-infectious diseases.https://doi.org/10.1038/s41746-024-01339-7 |
| spellingShingle | Fenglin Liu Zheng Li Qingyu Yin Jinfa Huang Jiebo Luo Anshul Thakur Kim Branson Patrick Schwab Bing Yin Xian Wu Yefeng Zheng David A. Clifton A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis npj Digital Medicine |
| title | A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis |
| title_full | A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis |
| title_fullStr | A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis |
| title_full_unstemmed | A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis |
| title_short | A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis |
| title_sort | multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis |
| url | https://doi.org/10.1038/s41746-024-01339-7 |
| work_keys_str_mv | AT fenglinliu amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT zhengli amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT qingyuyin amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT jinfahuang amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT jieboluo amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT anshulthakur amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT kimbranson amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT patrickschwab amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT bingyin amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT xianwu amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT yefengzheng amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT davidaclifton amultimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT fenglinliu multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT zhengli multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT qingyuyin multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT jinfahuang multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT jieboluo multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT anshulthakur multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT kimbranson multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT patrickschwab multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT bingyin multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT xianwu multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT yefengzheng multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis AT davidaclifton multimodalmultidomainmultilingualmedicalfoundationmodelforzeroshotclinicaldiagnosis |