Revealing Patient Dissatisfaction With Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis
BackgroundAccurately measuring the health care needs of patients with different diseases remains a public health challenge for health care management worldwide. There is a need for new computational methods to be able to assess the health care resources required by patients w...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-03-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2025/1/e66344 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849774280509751296 |
|---|---|
| author | Jiaxuan Li Yunchu Yang Chao Mao Patrick Cheong-Iao Pang Quanjing Zhu Dejian Xu Yapeng Wang |
| author_facet | Jiaxuan Li Yunchu Yang Chao Mao Patrick Cheong-Iao Pang Quanjing Zhu Dejian Xu Yapeng Wang |
| author_sort | Jiaxuan Li |
| collection | DOAJ |
| description |
BackgroundAccurately measuring the health care needs of patients with different diseases remains a public health challenge for health care management worldwide. There is a need for new computational methods to be able to assess the health care resources required by patients with different diseases to avoid wasting resources.
ObjectiveThis study aimed to assessing dissatisfaction with allocation of health care resources from the perspective of patients with different diseases that can help optimize resource allocation and better achieve several of the Sustainable Development Goals (SDGs), such as SDG 3 (“Good Health and Well-being”). Our goal was to show the effectiveness and practicality of large language models (LLMs) in assessing the distribution of health care resources.
MethodsWe used aspect-based sentiment analysis (ABSA), which can divide textual data into several aspects for sentiment analysis. In this study, we used Chat Generative Pretrained Transformer (ChatGPT) to perform ABSA of patient reviews based on 3 aspects (patient experience, physician skills and efficiency, and infrastructure and administration)00 in which we embedded chain-of-thought (CoT) prompting and compared the performance of Chinese and English LLMs on a Chinese dataset. Additionally, we used the International Classification of Diseases 11th Revision (ICD-11) application programming interface (API) to classify the sentiment analysis results into different disease categories.
ResultsWe evaluated the performance of the models by comparing predicted sentiments (either positive or negative) with the labels judged by human evaluators in terms of the aforementioned 3 aspects. The results showed that ChatGPT 3.5 is superior in a combination of stability, expense, and runtime considerations compared to ChatGPT-4o and Qwen-7b. The weighted total precision of our method based on the ABSA of patient reviews was 0.907, while the average accuracy of all 3 sampling methods was 0.893. Both values suggested that the model was able to achieve our objective. Using our approach, we identified that dissatisfaction is highest for sex-related diseases and lowest for circulatory diseases and that the need for better infrastructure and administration is much higher for blood-related diseases than for other diseases in China.
ConclusionsThe results prove that our method with LLMs can use patient reviews and the ICD-11 classification to assess the health care needs of patients with different diseases, which can assist with resource allocation rationally. |
| format | Article |
| id | doaj-art-01aa91d7ba9c48e493d7dc5f7b1ca449 |
| institution | DOAJ |
| issn | 1438-8871 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj-art-01aa91d7ba9c48e493d7dc5f7b1ca4492025-08-20T03:01:46ZengJMIR PublicationsJournal of Medical Internet Research1438-88712025-03-0127e6634410.2196/66344Revealing Patient Dissatisfaction With Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment AnalysisJiaxuan Lihttps://orcid.org/0000-0001-7647-6839Yunchu Yanghttps://orcid.org/0009-0009-1688-5695Chao Maohttps://orcid.org/0009-0008-0319-4687Patrick Cheong-Iao Panghttps://orcid.org/0000-0002-8820-5443Quanjing Zhuhttps://orcid.org/0000-0003-4813-0894Dejian Xuhttps://orcid.org/0009-0003-1528-5862Yapeng Wanghttps://orcid.org/0000-0002-1085-5091 BackgroundAccurately measuring the health care needs of patients with different diseases remains a public health challenge for health care management worldwide. There is a need for new computational methods to be able to assess the health care resources required by patients with different diseases to avoid wasting resources. ObjectiveThis study aimed to assessing dissatisfaction with allocation of health care resources from the perspective of patients with different diseases that can help optimize resource allocation and better achieve several of the Sustainable Development Goals (SDGs), such as SDG 3 (“Good Health and Well-being”). Our goal was to show the effectiveness and practicality of large language models (LLMs) in assessing the distribution of health care resources. MethodsWe used aspect-based sentiment analysis (ABSA), which can divide textual data into several aspects for sentiment analysis. In this study, we used Chat Generative Pretrained Transformer (ChatGPT) to perform ABSA of patient reviews based on 3 aspects (patient experience, physician skills and efficiency, and infrastructure and administration)00 in which we embedded chain-of-thought (CoT) prompting and compared the performance of Chinese and English LLMs on a Chinese dataset. Additionally, we used the International Classification of Diseases 11th Revision (ICD-11) application programming interface (API) to classify the sentiment analysis results into different disease categories. ResultsWe evaluated the performance of the models by comparing predicted sentiments (either positive or negative) with the labels judged by human evaluators in terms of the aforementioned 3 aspects. The results showed that ChatGPT 3.5 is superior in a combination of stability, expense, and runtime considerations compared to ChatGPT-4o and Qwen-7b. The weighted total precision of our method based on the ABSA of patient reviews was 0.907, while the average accuracy of all 3 sampling methods was 0.893. Both values suggested that the model was able to achieve our objective. Using our approach, we identified that dissatisfaction is highest for sex-related diseases and lowest for circulatory diseases and that the need for better infrastructure and administration is much higher for blood-related diseases than for other diseases in China. ConclusionsThe results prove that our method with LLMs can use patient reviews and the ICD-11 classification to assess the health care needs of patients with different diseases, which can assist with resource allocation rationally.https://www.jmir.org/2025/1/e66344 |
| spellingShingle | Jiaxuan Li Yunchu Yang Chao Mao Patrick Cheong-Iao Pang Quanjing Zhu Dejian Xu Yapeng Wang Revealing Patient Dissatisfaction With Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis Journal of Medical Internet Research |
| title | Revealing Patient Dissatisfaction With Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis |
| title_full | Revealing Patient Dissatisfaction With Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis |
| title_fullStr | Revealing Patient Dissatisfaction With Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis |
| title_full_unstemmed | Revealing Patient Dissatisfaction With Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis |
| title_short | Revealing Patient Dissatisfaction With Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis |
| title_sort | revealing patient dissatisfaction with health care resource allocation in multiple dimensions using large language models and the international classification of diseases 11th revision aspect based sentiment analysis |
| url | https://www.jmir.org/2025/1/e66344 |
| work_keys_str_mv | AT jiaxuanli revealingpatientdissatisfactionwithhealthcareresourceallocationinmultipledimensionsusinglargelanguagemodelsandtheinternationalclassificationofdiseases11threvisionaspectbasedsentimentanalysis AT yunchuyang revealingpatientdissatisfactionwithhealthcareresourceallocationinmultipledimensionsusinglargelanguagemodelsandtheinternationalclassificationofdiseases11threvisionaspectbasedsentimentanalysis AT chaomao revealingpatientdissatisfactionwithhealthcareresourceallocationinmultipledimensionsusinglargelanguagemodelsandtheinternationalclassificationofdiseases11threvisionaspectbasedsentimentanalysis AT patrickcheongiaopang revealingpatientdissatisfactionwithhealthcareresourceallocationinmultipledimensionsusinglargelanguagemodelsandtheinternationalclassificationofdiseases11threvisionaspectbasedsentimentanalysis AT quanjingzhu revealingpatientdissatisfactionwithhealthcareresourceallocationinmultipledimensionsusinglargelanguagemodelsandtheinternationalclassificationofdiseases11threvisionaspectbasedsentimentanalysis AT dejianxu revealingpatientdissatisfactionwithhealthcareresourceallocationinmultipledimensionsusinglargelanguagemodelsandtheinternationalclassificationofdiseases11threvisionaspectbasedsentimentanalysis AT yapengwang revealingpatientdissatisfactionwithhealthcareresourceallocationinmultipledimensionsusinglargelanguagemodelsandtheinternationalclassificationofdiseases11threvisionaspectbasedsentimentanalysis |