Performance of popular large language models in glaucoma patient education: A randomized controlled study
Purpose: The advent of chatbots based on large language models (LLMs), such as ChatGPT, has significantly transformed knowledge acquisition. However, the application of LLMs in glaucoma patient education remains elusive. In this study, we comprehensively compared the performance of four common LLMs...
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-05-01
|
| Series: | Advances in Ophthalmology Practice and Research |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2667376224000738 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849761717089730560 |
|---|---|
| author | Yuyu Cao Wei Lu Runhan Shi Fuying Liu Steven Liu Xinwei Xu Jin Yang Guangyu Rong Changchang Xin Xujiao Zhou Xinghuai Sun Jiaxu Hong |
| author_facet | Yuyu Cao Wei Lu Runhan Shi Fuying Liu Steven Liu Xinwei Xu Jin Yang Guangyu Rong Changchang Xin Xujiao Zhou Xinghuai Sun Jiaxu Hong |
| author_sort | Yuyu Cao |
| collection | DOAJ |
| description | Purpose: The advent of chatbots based on large language models (LLMs), such as ChatGPT, has significantly transformed knowledge acquisition. However, the application of LLMs in glaucoma patient education remains elusive. In this study, we comprehensively compared the performance of four common LLMs – Qwen, Baichuan 2, ChatGPT-4o, and PaLM 2 – in the context of glaucoma patient education. Methods: Initially, senior ophthalmologists were asked with scoring responses generated by the LLMs, which were answers to the most frequent glaucoma-related questions posed by patients. The Chinese Readability Platform was employed to assess the recommended reading age and reading difficulty score of the four LLMs. Subsequently, optimized models were filtered, and 29 glaucoma patients participated in posing questions to the chatbots and scoring the answers within a real-world clinical setting. Attending ophthalmologists were also required to score the answers across five dimensions: correctness, completeness, readability, helpfulness, and safety. Patients, on the other hand, scored the answers based on three dimensions: satisfaction, readability, and helpfulness. Results: In the first stage, Baichuan 2 and ChatGPT-4o outperformed the other two models, though ChatGPT-4o had higher recommended reading age and reading difficulty scores. In the second stage, both Baichuan 2 and ChatGPT-4o demonstrated exceptional performance among patients and ophthalmologists, with no statistically significant differences observed. Conclusions: Our research identifies Baichuan 2 and ChatGPT-4o as prominent LLMs, offering viable options for glaucoma education. |
| format | Article |
| id | doaj-art-c83d2c79a8ea43deab583aa0b4088353 |
| institution | DOAJ |
| issn | 2667-3762 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Advances in Ophthalmology Practice and Research |
| spelling | doaj-art-c83d2c79a8ea43deab583aa0b40883532025-08-20T03:05:56ZengElsevierAdvances in Ophthalmology Practice and Research2667-37622025-05-0152889410.1016/j.aopr.2024.12.002Performance of popular large language models in glaucoma patient education: A randomized controlled studyYuyu Cao0Wei Lu1Runhan Shi2Fuying Liu3Steven Liu4Xinwei Xu5Jin Yang6Guangyu Rong7Changchang Xin8Xujiao Zhou9Xinghuai Sun10Jiaxu Hong11Department of Ophthalmology, Eye & ENT Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai, China; NHC Key Laboratory of Myopia and Related Eye Diseases Shanghai, China; Shanghai Engineering Research Center of Synthetic Immunology, Shanghai, China; Department of Ophthalmology, Children's Hospital of Fudan University, National Pediatric Medical Center of China, Shanghai, ChinaDepartment of Ophthalmology and Vision Science, Shanghai Eye Ear Nose and Throat Hospital, Fudan University, Shanghai, ChinaDepartment of Ophthalmology, Eye & ENT Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai, China; NHC Key Laboratory of Myopia and Related Eye Diseases Shanghai, China; Shanghai Engineering Research Center of Synthetic Immunology, Shanghai, China; Department of Ophthalmology, Children's Hospital of Fudan University, National Pediatric Medical Center of China, Shanghai, ChinaDepartment of Ophthalmology and Vision Science, Shanghai Eye Ear Nose and Throat Hospital, Fudan University, Shanghai, China; People‘s Hospital of Junan, Qingdao University, Shandong, ChinaDepartment of Statistics, College of Liberal Arts & Sciences, University of Illinois Urbana-Champaign, Urbana, Champaign, USAFaculty of Business and Economics, Hong Kong University, Hong Kong, ChinaDepartment of Ophthalmology and Vision Science, Shanghai Eye Ear Nose and Throat Hospital, Fudan University, Shanghai, ChinaDepartment of Ophthalmology, Eye & ENT Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai, China; NHC Key Laboratory of Myopia and Related Eye Diseases Shanghai, China; Shanghai Engineering Research Center of Synthetic Immunology, Shanghai, China; Department of Ophthalmology, Children's Hospital of Fudan University, National Pediatric Medical Center of China, Shanghai, ChinaDepartment of Ophthalmology, Eye & ENT Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai, China; NHC Key Laboratory of Myopia and Related Eye Diseases Shanghai, China; Shanghai Engineering Research Center of Synthetic Immunology, Shanghai, China; Department of Ophthalmology, Children's Hospital of Fudan University, National Pediatric Medical Center of China, Shanghai, ChinaDepartment of Ophthalmology, Eye & ENT Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai, China; NHC Key Laboratory of Myopia and Related Eye Diseases Shanghai, China; Shanghai Engineering Research Center of Synthetic Immunology, Shanghai, China; Department of Ophthalmology, Children's Hospital of Fudan University, National Pediatric Medical Center of China, Shanghai, ChinaDepartment of Ophthalmology and Vision Science, Shanghai Eye Ear Nose and Throat Hospital, Fudan University, Shanghai, China; Corresponding author.Department of Ophthalmology, Eye & ENT Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai, China; NHC Key Laboratory of Myopia and Related Eye Diseases Shanghai, China; Shanghai Engineering Research Center of Synthetic Immunology, Shanghai, China; Department of Ophthalmology, Children's Hospital of Fudan University, National Pediatric Medical Center of China, Shanghai, China; Corresponding author. Department of Ophthalmology, Eye & ENT Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai, 200031, China.Purpose: The advent of chatbots based on large language models (LLMs), such as ChatGPT, has significantly transformed knowledge acquisition. However, the application of LLMs in glaucoma patient education remains elusive. In this study, we comprehensively compared the performance of four common LLMs – Qwen, Baichuan 2, ChatGPT-4o, and PaLM 2 – in the context of glaucoma patient education. Methods: Initially, senior ophthalmologists were asked with scoring responses generated by the LLMs, which were answers to the most frequent glaucoma-related questions posed by patients. The Chinese Readability Platform was employed to assess the recommended reading age and reading difficulty score of the four LLMs. Subsequently, optimized models were filtered, and 29 glaucoma patients participated in posing questions to the chatbots and scoring the answers within a real-world clinical setting. Attending ophthalmologists were also required to score the answers across five dimensions: correctness, completeness, readability, helpfulness, and safety. Patients, on the other hand, scored the answers based on three dimensions: satisfaction, readability, and helpfulness. Results: In the first stage, Baichuan 2 and ChatGPT-4o outperformed the other two models, though ChatGPT-4o had higher recommended reading age and reading difficulty scores. In the second stage, both Baichuan 2 and ChatGPT-4o demonstrated exceptional performance among patients and ophthalmologists, with no statistically significant differences observed. Conclusions: Our research identifies Baichuan 2 and ChatGPT-4o as prominent LLMs, offering viable options for glaucoma education.http://www.sciencedirect.com/science/article/pii/S2667376224000738GlaucomaPatient educationLarge language modelsChatGPTPaLMBaichuan |
| spellingShingle | Yuyu Cao Wei Lu Runhan Shi Fuying Liu Steven Liu Xinwei Xu Jin Yang Guangyu Rong Changchang Xin Xujiao Zhou Xinghuai Sun Jiaxu Hong Performance of popular large language models in glaucoma patient education: A randomized controlled study Advances in Ophthalmology Practice and Research Glaucoma Patient education Large language models ChatGPT PaLM Baichuan |
| title | Performance of popular large language models in glaucoma patient education: A randomized controlled study |
| title_full | Performance of popular large language models in glaucoma patient education: A randomized controlled study |
| title_fullStr | Performance of popular large language models in glaucoma patient education: A randomized controlled study |
| title_full_unstemmed | Performance of popular large language models in glaucoma patient education: A randomized controlled study |
| title_short | Performance of popular large language models in glaucoma patient education: A randomized controlled study |
| title_sort | performance of popular large language models in glaucoma patient education a randomized controlled study |
| topic | Glaucoma Patient education Large language models ChatGPT PaLM Baichuan |
| url | http://www.sciencedirect.com/science/article/pii/S2667376224000738 |
| work_keys_str_mv | AT yuyucao performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT weilu performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT runhanshi performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT fuyingliu performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT stevenliu performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT xinweixu performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT jinyang performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT guangyurong performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT changchangxin performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT xujiaozhou performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT xinghuaisun performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy AT jiaxuhong performanceofpopularlargelanguagemodelsinglaucomapatienteducationarandomizedcontrolledstudy |