A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings
Abstract Large foundation models show promise in biomedicine but face challenges in clinical use due to performance gaps, accessibility, cost, and lack of scalable evaluation. Here we show that open-source small multimodal models can bridge these gaps in radiology by generating free-text findings fr...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-04-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-025-58344-x |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849765447399899136 |
|---|---|
| author | Juan Manuel Zambrano Chaves Shih-Cheng Huang Yanbo Xu Hanwen Xu Naoto Usuyama Sheng Zhang Fei Wang Yujia Xie Mahmoud Khademi Ziyi Yang Hany Awadalla Julia Gong Houdong Hu Jianwei Yang Chunyuan Li Jianfeng Gao Yu Gu Cliff Wong Mu Wei Tristan Naumann Muhao Chen Matthew P. Lungren Akshay Chaudhari Serena Yeung-Levy Curtis P. Langlotz Sheng Wang Hoifung Poon |
| author_facet | Juan Manuel Zambrano Chaves Shih-Cheng Huang Yanbo Xu Hanwen Xu Naoto Usuyama Sheng Zhang Fei Wang Yujia Xie Mahmoud Khademi Ziyi Yang Hany Awadalla Julia Gong Houdong Hu Jianwei Yang Chunyuan Li Jianfeng Gao Yu Gu Cliff Wong Mu Wei Tristan Naumann Muhao Chen Matthew P. Lungren Akshay Chaudhari Serena Yeung-Levy Curtis P. Langlotz Sheng Wang Hoifung Poon |
| author_sort | Juan Manuel Zambrano Chaves |
| collection | DOAJ |
| description | Abstract Large foundation models show promise in biomedicine but face challenges in clinical use due to performance gaps, accessibility, cost, and lack of scalable evaluation. Here we show that open-source small multimodal models can bridge these gaps in radiology by generating free-text findings from chest X-ray images. Our data-centric approach leverages 697K curated radiology image-text pairs to train a specialized, domain-adapted chest X-ray encoder. We integrate this encoder with pre-trained language models via a lightweight adapter that aligns image and text modalities. To enable robust, clinically relevant evaluation, we develop and validate CheXprompt, a GPT-4-based metric for assessing factual accuracy aligned with radiologists’ evaluations. Benchmarked with CheXprompt and other standard factuality metrics, LLaVA-Rad (7B) achieves state-of-the-art performance, outperforming much larger models like GPT-4V and Med-PaLM M (84B). While not immediately ready for real-time clinical deployment, LLaVA-Rad is a scalable, privacy-preserving and cost-effective step towards clinically adaptable multimodal AI for radiology. |
| format | Article |
| id | doaj-art-ca3805a3f16c41a8a388f41c7936fd5a |
| institution | DOAJ |
| issn | 2041-1723 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj-art-ca3805a3f16c41a8a388f41c7936fd5a2025-08-20T03:04:51ZengNature PortfolioNature Communications2041-17232025-04-0116111510.1038/s41467-025-58344-xA clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findingsJuan Manuel Zambrano Chaves0Shih-Cheng Huang1Yanbo Xu2Hanwen Xu3Naoto Usuyama4Sheng Zhang5Fei Wang6Yujia Xie7Mahmoud Khademi8Ziyi Yang9Hany Awadalla10Julia Gong11Houdong Hu12Jianwei Yang13Chunyuan Li14Jianfeng Gao15Yu Gu16Cliff Wong17Mu Wei18Tristan Naumann19Muhao Chen20Matthew P. Lungren21Akshay Chaudhari22Serena Yeung-Levy23Curtis P. Langlotz24Sheng Wang25Hoifung Poon26Microsoft ResearchStanford UniversityMicrosoft ResearchUniversity of WashingtonMicrosoft ResearchMicrosoft ResearchUniversity of Southern CaliforniaMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchMicrosoft ResearchUniversity of CaliforniaMicrosoft ResearchStanford UniversityStanford UniversityStanford UniversityUniversity of WashingtonMicrosoft ResearchAbstract Large foundation models show promise in biomedicine but face challenges in clinical use due to performance gaps, accessibility, cost, and lack of scalable evaluation. Here we show that open-source small multimodal models can bridge these gaps in radiology by generating free-text findings from chest X-ray images. Our data-centric approach leverages 697K curated radiology image-text pairs to train a specialized, domain-adapted chest X-ray encoder. We integrate this encoder with pre-trained language models via a lightweight adapter that aligns image and text modalities. To enable robust, clinically relevant evaluation, we develop and validate CheXprompt, a GPT-4-based metric for assessing factual accuracy aligned with radiologists’ evaluations. Benchmarked with CheXprompt and other standard factuality metrics, LLaVA-Rad (7B) achieves state-of-the-art performance, outperforming much larger models like GPT-4V and Med-PaLM M (84B). While not immediately ready for real-time clinical deployment, LLaVA-Rad is a scalable, privacy-preserving and cost-effective step towards clinically adaptable multimodal AI for radiology.https://doi.org/10.1038/s41467-025-58344-x |
| spellingShingle | Juan Manuel Zambrano Chaves Shih-Cheng Huang Yanbo Xu Hanwen Xu Naoto Usuyama Sheng Zhang Fei Wang Yujia Xie Mahmoud Khademi Ziyi Yang Hany Awadalla Julia Gong Houdong Hu Jianwei Yang Chunyuan Li Jianfeng Gao Yu Gu Cliff Wong Mu Wei Tristan Naumann Muhao Chen Matthew P. Lungren Akshay Chaudhari Serena Yeung-Levy Curtis P. Langlotz Sheng Wang Hoifung Poon A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings Nature Communications |
| title | A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings |
| title_full | A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings |
| title_fullStr | A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings |
| title_full_unstemmed | A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings |
| title_short | A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings |
| title_sort | clinically accessible small multimodal radiology model and evaluation metric for chest x ray findings |
| url | https://doi.org/10.1038/s41467-025-58344-x |
| work_keys_str_mv | AT juanmanuelzambranochaves aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT shihchenghuang aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT yanboxu aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT hanwenxu aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT naotousuyama aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT shengzhang aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT feiwang aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT yujiaxie aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT mahmoudkhademi aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT ziyiyang aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT hanyawadalla aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT juliagong aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT houdonghu aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT jianweiyang aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT chunyuanli aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT jianfenggao aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT yugu aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT cliffwong aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT muwei aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT tristannaumann aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT muhaochen aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT matthewplungren aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT akshaychaudhari aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT serenayeunglevy aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT curtisplanglotz aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT shengwang aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT hoifungpoon aclinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT juanmanuelzambranochaves clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT shihchenghuang clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT yanboxu clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT hanwenxu clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT naotousuyama clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT shengzhang clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT feiwang clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT yujiaxie clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT mahmoudkhademi clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT ziyiyang clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT hanyawadalla clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT juliagong clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT houdonghu clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT jianweiyang clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT chunyuanli clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT jianfenggao clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT yugu clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT cliffwong clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT muwei clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT tristannaumann clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT muhaochen clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT matthewplungren clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT akshaychaudhari clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT serenayeunglevy clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT curtisplanglotz clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT shengwang clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings AT hoifungpoon clinicallyaccessiblesmallmultimodalradiologymodelandevaluationmetricforchestxrayfindings |