Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms
Abstract Faced with challenging cases, doctors are increasingly seeking diagnostic advice from large language models (LLMs). This study aims to compare the ability of LLMs and human physicians to diagnose challenging cases. An offline dataset of 67 challenging cases with primary gastrointestinal sym...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-02-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-025-01486-5 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823861540954046464 |
|---|---|
| author | Xintian Yang Tongxin Li Han Wang Rongchun Zhang Zhi Ni Na Liu Huihong Zhai Jianghai Zhao Fandong Meng Zhongyin Zhou Shanhong Tang Limei Wang Xiangping Wang Hui Luo Gui Ren Linhui Zhang Xiaoyu Kang Jun Wang Ning Bo Xiaoning Yang Weijie Xue Xiaoyin Zhang Ning Chen Rui Guo Baiwen Li Yajun Li Yaling Liu Tiantian Zhang Shuhui Liang Yong Lv Yongzhan Nie Daiming Fan Lina Zhao Yanglin Pan |
| author_facet | Xintian Yang Tongxin Li Han Wang Rongchun Zhang Zhi Ni Na Liu Huihong Zhai Jianghai Zhao Fandong Meng Zhongyin Zhou Shanhong Tang Limei Wang Xiangping Wang Hui Luo Gui Ren Linhui Zhang Xiaoyu Kang Jun Wang Ning Bo Xiaoning Yang Weijie Xue Xiaoyin Zhang Ning Chen Rui Guo Baiwen Li Yajun Li Yaling Liu Tiantian Zhang Shuhui Liang Yong Lv Yongzhan Nie Daiming Fan Lina Zhao Yanglin Pan |
| author_sort | Xintian Yang |
| collection | DOAJ |
| description | Abstract Faced with challenging cases, doctors are increasingly seeking diagnostic advice from large language models (LLMs). This study aims to compare the ability of LLMs and human physicians to diagnose challenging cases. An offline dataset of 67 challenging cases with primary gastrointestinal symptoms was used to solicit possible diagnoses from seven LLMs and 22 gastroenterologists. The diagnoses by Claude 3.5 Sonnet covered the highest proportion (95% confidence interval [CI]) of instructive diagnoses (76.1%, [70.6%–80.9%]), significantly surpassing all the gastroenterologists (p < 0.05 for all). Claude 3.5 Sonnet achieved a significantly higher coverage rate (95% CI) than that of the gastroenterologists using search engines or other traditional resource (76.1% [70.6%–80.9%] vs. 45.5% [40.7%-50.4%], p < 0.001). The study highlights that advanced LLMs may assist gastroenterologists with instructive, time-saving, and cost-effective diagnostic scopes in challenging cases. |
| format | Article |
| id | doaj-art-32978c9a4f39481bacda086566ca7d43 |
| institution | Kabale University |
| issn | 2398-6352 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj-art-32978c9a4f39481bacda086566ca7d432025-02-09T12:55:36ZengNature Portfolionpj Digital Medicine2398-63522025-02-018111210.1038/s41746-025-01486-5Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptomsXintian Yang0Tongxin Li1Han Wang2Rongchun Zhang3Zhi Ni4Na Liu5Huihong Zhai6Jianghai Zhao7Fandong Meng8Zhongyin Zhou9Shanhong Tang10Limei Wang11Xiangping Wang12Hui Luo13Gui Ren14Linhui Zhang15Xiaoyu Kang16Jun Wang17Ning Bo18Xiaoning Yang19Weijie Xue20Xiaoyin Zhang21Ning Chen22Rui Guo23Baiwen Li24Yajun Li25Yaling Liu26Tiantian Zhang27Shuhui Liang28Yong Lv29Yongzhan Nie30Daiming Fan31Lina Zhao32Yanglin Pan33State Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityDepartment of Pathology, Union Hospital, Tongji Medical College, Huazhong University of Science and TechnologyDepartment of Gastroenterology, Xiamen Humanity Hospital, Fujian Medical UniversityDepartment of Gastroenterology, Xiamen Humanity Hospital, Fujian Medical UniversityDepartment of Gastroenterology, Hainan General Hospital (Hainan Affiliated Hospital of Hainan Medical University)Department of Gastroenterology, Xuanwu Hospital, Capital Medical UniversityDepartment of Gastroenterology, Huaihe Hospital of Henan UniversityDepartment of Gastroenterology, Beijing Friendship Hospital, Capital Medical University, National Clinical Research Center for Digestive Disease, Beijing Digestive Disease Center, Beijing Key Laboratory for Precancerous Lesion of Digestive DiseaseDepartment of Gastroenterology, Renmin Hospital of Wuhan UniversityDepartment of Gastroenterology, The General Hospital of Western Theater CommandDepartment of Gastroenterology, Shaanxi Second Provincial People’s HospitalState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityDepartment of Gastroenterology, The 986th Hospital of Xijing Hospital, Air Force Military Medical UniversityDepartment of Gastroenterology, The Second Affiliated Hospital of Chongqing Medical UniversityDepartment of Gastroenterology, Zhongshan Hospital of Xiamen University, School of Medicine, Xiamen UniversityDepartment of Transplantation and Pediatric Surgery, Kumamoto University HospitalDepartment of Gastroenterology, National Clinical Research Center of Infectious Disease, The Third People’s Hospital of Shenzhen, The Second Affiliated Hospital of Southern University of Science and TechnologyDepartment of Gastroenterology, Peking University People’s Hospital, Peking UniversityDepartment of Gastroenterology, Beijing Shijingshan Hospital, Capital Medical UniversityDepartment of Gastroenterology, Shanghai General Hospital, Shanghai Jiao Tong University School of MedicineDepartment of Gastroenterology, General Hospital of Ningxia Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityDepartment of Radiotherapy, Xijing Hospital, Fourth Military Medical UniversityState Key Laboratory of Holistic Integrative Management of Gastrointestinal Cancers and National Clinical Research Center for Digestive Diseases, Xijing Hospital of Digestive Diseases, Fourth Military Medical UniversityAbstract Faced with challenging cases, doctors are increasingly seeking diagnostic advice from large language models (LLMs). This study aims to compare the ability of LLMs and human physicians to diagnose challenging cases. An offline dataset of 67 challenging cases with primary gastrointestinal symptoms was used to solicit possible diagnoses from seven LLMs and 22 gastroenterologists. The diagnoses by Claude 3.5 Sonnet covered the highest proportion (95% confidence interval [CI]) of instructive diagnoses (76.1%, [70.6%–80.9%]), significantly surpassing all the gastroenterologists (p < 0.05 for all). Claude 3.5 Sonnet achieved a significantly higher coverage rate (95% CI) than that of the gastroenterologists using search engines or other traditional resource (76.1% [70.6%–80.9%] vs. 45.5% [40.7%-50.4%], p < 0.001). The study highlights that advanced LLMs may assist gastroenterologists with instructive, time-saving, and cost-effective diagnostic scopes in challenging cases.https://doi.org/10.1038/s41746-025-01486-5 |
| spellingShingle | Xintian Yang Tongxin Li Han Wang Rongchun Zhang Zhi Ni Na Liu Huihong Zhai Jianghai Zhao Fandong Meng Zhongyin Zhou Shanhong Tang Limei Wang Xiangping Wang Hui Luo Gui Ren Linhui Zhang Xiaoyu Kang Jun Wang Ning Bo Xiaoning Yang Weijie Xue Xiaoyin Zhang Ning Chen Rui Guo Baiwen Li Yajun Li Yaling Liu Tiantian Zhang Shuhui Liang Yong Lv Yongzhan Nie Daiming Fan Lina Zhao Yanglin Pan Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms npj Digital Medicine |
| title | Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms |
| title_full | Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms |
| title_fullStr | Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms |
| title_full_unstemmed | Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms |
| title_short | Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms |
| title_sort | multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms |
| url | https://doi.org/10.1038/s41746-025-01486-5 |
| work_keys_str_mv | AT xintianyang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT tongxinli multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT hanwang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT rongchunzhang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT zhini multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT naliu multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT huihongzhai multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT jianghaizhao multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT fandongmeng multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT zhongyinzhou multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT shanhongtang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT limeiwang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT xiangpingwang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT huiluo multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT guiren multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT linhuizhang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT xiaoyukang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT junwang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT ningbo multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT xiaoningyang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT weijiexue multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT xiaoyinzhang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT ningchen multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT ruiguo multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT baiwenli multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT yajunli multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT yalingliu multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT tiantianzhang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT shuhuiliang multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT yonglv multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT yongzhannie multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT daimingfan multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT linazhao multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms AT yanglinpan multiplelargelanguagemodelsversusexperiencedphysiciansindiagnosingchallengingcaseswithgastrointestinalsymptoms |