Language models for data extraction and risk of bias assessment in complementary medicine
Abstract Large language models (LLMs) have the potential to enhance evidence synthesis efficiency and accuracy. This study assessed LLM-only and LLM-assisted methods in data extraction and risk of bias assessment for 107 trials on complementary medicine. Moonshot-v1-128k and Claude-3.5-sonnet achiev...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-01-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-025-01457-w |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832571346545016832 |
|---|---|
| author | Honghao Lai Jiayi Liu Chunyang Bai Hui Liu Bei Pan Xufei Luo Liangying Hou Weilong Zhao Danni Xia Jinhui Tian Yaolong Chen Lu Zhang Janne Estill Jie Liu Xing Liao Nannan Shi Xin Sun Hongcai Shang Zhaoxiang Bian Kehu Yang Luqi Huang Long Ge On behalf of ADVANCED Working Group |
| author_facet | Honghao Lai Jiayi Liu Chunyang Bai Hui Liu Bei Pan Xufei Luo Liangying Hou Weilong Zhao Danni Xia Jinhui Tian Yaolong Chen Lu Zhang Janne Estill Jie Liu Xing Liao Nannan Shi Xin Sun Hongcai Shang Zhaoxiang Bian Kehu Yang Luqi Huang Long Ge On behalf of ADVANCED Working Group |
| author_sort | Honghao Lai |
| collection | DOAJ |
| description | Abstract Large language models (LLMs) have the potential to enhance evidence synthesis efficiency and accuracy. This study assessed LLM-only and LLM-assisted methods in data extraction and risk of bias assessment for 107 trials on complementary medicine. Moonshot-v1-128k and Claude-3.5-sonnet achieved high accuracy (≥95%), with LLM-assisted methods performing better (≥97%). LLM-assisted methods significantly reduced processing time (14.7 and 5.9 min vs. 86.9 and 10.4 min for conventional methods). These findings highlight LLMs’ potential when integrated with human expertise. |
| format | Article |
| id | doaj-art-f801c218a31e477e9dab93f0fc38be67 |
| institution | Kabale University |
| issn | 2398-6352 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj-art-f801c218a31e477e9dab93f0fc38be672025-02-02T12:43:42ZengNature Portfolionpj Digital Medicine2398-63522025-01-01811810.1038/s41746-025-01457-wLanguage models for data extraction and risk of bias assessment in complementary medicineHonghao Lai0Jiayi Liu1Chunyang Bai2Hui Liu3Bei Pan4Xufei Luo5Liangying Hou6Weilong Zhao7Danni Xia8Jinhui Tian9Yaolong Chen10Lu Zhang11Janne Estill12Jie Liu13Xing Liao14Nannan Shi15Xin Sun16Hongcai Shang17Zhaoxiang Bian18Kehu Yang19Luqi Huang20Long Ge21On behalf of ADVANCED Working GroupDepartment of Health Policy and Health Management, School of Public Health, Lanzhou UniversityDepartment of Health Policy and Health Management, School of Public Health, Lanzhou UniversitySchool of Nursing, Southern Medical UniversityEvidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou UniversityEvidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou UniversityEvidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou UniversityEvidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou UniversityDepartment of Health Policy and Health Management, School of Public Health, Lanzhou UniversityDepartment of Health Policy and Health Management, School of Public Health, Lanzhou UniversityEvidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou UniversityEvidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou UniversityDepartment of Computer Science, Hong Kong Baptist UniversityEvidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou UniversityDepartment of Oncology, Guang’ anmen Hospital, China Academy of Chinese Medical SciencesInstitute of Basic Research of Clinical Medicine, China Academy of Chinese Medical SciencesInstitute of Basic Research of Clinical Medicine, China Academy of Chinese Medical SciencesChinese Evidence-Based Medicine Center, West China Hospital, Sichuan UniversityDongzhimen Hospital, Beijing University of Chinese MedicineSchool of Chinese Medicine, Hong Kong Baptist UniversityEvidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou UniversityChina Center for Evidence Based Traditional Chinese Medicine, China Academy of Chinese Medical SciencesDepartment of Health Policy and Health Management, School of Public Health, Lanzhou UniversityAbstract Large language models (LLMs) have the potential to enhance evidence synthesis efficiency and accuracy. This study assessed LLM-only and LLM-assisted methods in data extraction and risk of bias assessment for 107 trials on complementary medicine. Moonshot-v1-128k and Claude-3.5-sonnet achieved high accuracy (≥95%), with LLM-assisted methods performing better (≥97%). LLM-assisted methods significantly reduced processing time (14.7 and 5.9 min vs. 86.9 and 10.4 min for conventional methods). These findings highlight LLMs’ potential when integrated with human expertise.https://doi.org/10.1038/s41746-025-01457-w |
| spellingShingle | Honghao Lai Jiayi Liu Chunyang Bai Hui Liu Bei Pan Xufei Luo Liangying Hou Weilong Zhao Danni Xia Jinhui Tian Yaolong Chen Lu Zhang Janne Estill Jie Liu Xing Liao Nannan Shi Xin Sun Hongcai Shang Zhaoxiang Bian Kehu Yang Luqi Huang Long Ge On behalf of ADVANCED Working Group Language models for data extraction and risk of bias assessment in complementary medicine npj Digital Medicine |
| title | Language models for data extraction and risk of bias assessment in complementary medicine |
| title_full | Language models for data extraction and risk of bias assessment in complementary medicine |
| title_fullStr | Language models for data extraction and risk of bias assessment in complementary medicine |
| title_full_unstemmed | Language models for data extraction and risk of bias assessment in complementary medicine |
| title_short | Language models for data extraction and risk of bias assessment in complementary medicine |
| title_sort | language models for data extraction and risk of bias assessment in complementary medicine |
| url | https://doi.org/10.1038/s41746-025-01457-w |
| work_keys_str_mv | AT honghaolai languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT jiayiliu languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT chunyangbai languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT huiliu languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT beipan languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT xufeiluo languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT liangyinghou languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT weilongzhao languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT dannixia languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT jinhuitian languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT yaolongchen languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT luzhang languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT janneestill languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT jieliu languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT xingliao languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT nannanshi languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT xinsun languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT hongcaishang languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT zhaoxiangbian languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT kehuyang languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT luqihuang languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT longge languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine AT onbehalfofadvancedworkinggroup languagemodelsfordataextractionandriskofbiasassessmentincomplementarymedicine |