Large Language Models and Artificial Neural Networks for Assessing 1-Year Mortality in Patients With Myocardial Infarction: Analysis From the Medical Information Mart for Intensive Care IV (MIMIC-IV) Database
BackgroundAccurate mortality risk prediction is crucial for effective cardiovascular risk management. Recent advancements in artificial intelligence (AI) have demonstrated potential in this specific medical field. Qwen-2 and Llama-3 are high-performance, open-source large lan...
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-05-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2025/1/e67253 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850032301038108672 |
|---|---|
| author | Boqun Shi Liangguo Chen Shuo Pang Yue Wang Shen Wang Fadong Li Wenxin Zhao Pengrong Guo Leli Zhang Chu Fan Yi Zou Xiaofan Wu |
| author_facet | Boqun Shi Liangguo Chen Shuo Pang Yue Wang Shen Wang Fadong Li Wenxin Zhao Pengrong Guo Leli Zhang Chu Fan Yi Zou Xiaofan Wu |
| author_sort | Boqun Shi |
| collection | DOAJ |
| description |
BackgroundAccurate mortality risk prediction is crucial for effective cardiovascular risk management. Recent advancements in artificial intelligence (AI) have demonstrated potential in this specific medical field. Qwen-2 and Llama-3 are high-performance, open-source large language models (LLMs) available online. An artificial neural network (ANN) algorithm derived from the SWEDEHEART (Swedish Web System for Enhancement and Development of Evidence-Based Care in Heart Disease Evaluated According to Recommended Therapies) registry, termed SWEDEHEART-AI, can predict patient prognosis following acute myocardial infarction (AMI).
ObjectiveThis study aims to evaluate the 3 models mentioned above in predicting 1-year all-cause mortality in critically ill patients with AMI.
MethodsThe Medical Information Mart for Intensive Care IV (MIMIC-IV) database is a publicly available data set in critical care medicine. We included 2758 patients who were first admitted for AMI and discharged alive. SWEDEHEART-AI calculated the mortality rate based on each patient’s 21 clinical variables. Qwen-2 and Llama-3 analyzed the content of patients’ discharge records and directly provided a 1-decimal value between 0 and 1 to represent 1-year death risk probabilities. The patients’ actual mortality was verified using follow-up data. The predictive performance of the 3 models was assessed and compared using the Harrell C-statistic (C-index), the area under the receiver operating characteristic curve (AUROC), calibration plots, Kaplan-Meier curves, and decision curve analysis.
ResultsSWEDEHEART-AI demonstrated strong discrimination in predicting 1-year all-cause mortality in patients with AMI, with a higher C-index than Qwen-2 and Llama-3 (C-index 0.72, 95% CI 0.69-0.74 vs C-index 0.65, 0.62-0.67 vs C-index 0.56, 95% CI 0.53-0.58, respectively; all P<.001 for both comparisons). SWEDEHEART-AI also showed high and consistent AUROC in the time-dependent ROC curve. The death rates calculated by SWEDEHEART-AI were positively correlated with actual mortality, and the 3 risk classes derived from this model showed clear differentiation in the Kaplan-Meier curve (P<.001). Calibration plots indicated that SWEDEHEART-AI tended to overestimate mortality risk, with an observed-to-expected ratio of 0.478. Compared with the LLMs, SWEDEHEART-AI demonstrated positive and greater net benefits at risk thresholds below 19%.
ConclusionsSWEDEHEART-AI, a trained ANN model, demonstrated the best performance, with strong discrimination and clinical utility in predicting 1-year all-cause mortality in patients with AMI from an intensive care cohort. Among the LLMs, Qwen-2 outperformed Llama-3 and showed moderate predictive value. Qwen-2 and SWEDEHEART-AI exhibited comparable classification effectiveness. The future integration of LLMs into clinical decision support systems holds promise for accurate risk stratification in patients with AMI; however, further research is needed to optimize LLM performance and address calibration issues across diverse patient populations. |
| format | Article |
| id | doaj-art-2a473ecfba09441884c5b3db864da5c8 |
| institution | DOAJ |
| issn | 1438-8871 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj-art-2a473ecfba09441884c5b3db864da5c82025-08-20T02:58:41ZengJMIR PublicationsJournal of Medical Internet Research1438-88712025-05-0127e6725310.2196/67253Large Language Models and Artificial Neural Networks for Assessing 1-Year Mortality in Patients With Myocardial Infarction: Analysis From the Medical Information Mart for Intensive Care IV (MIMIC-IV) DatabaseBoqun Shihttps://orcid.org/0000-0002-3403-9128Liangguo Chenhttps://orcid.org/0009-0008-4276-9068Shuo Panghttps://orcid.org/0009-0007-9817-2327Yue Wanghttps://orcid.org/0000-0001-7676-3846Shen Wanghttps://orcid.org/0000-0002-7321-6211Fadong Lihttps://orcid.org/0009-0003-5946-3018Wenxin Zhaohttps://orcid.org/0009-0009-4842-3292Pengrong Guohttps://orcid.org/0000-0002-5944-3194Leli Zhanghttps://orcid.org/0009-0004-8024-2161Chu Fanhttps://orcid.org/0009-0005-0567-7883Yi Zouhttps://orcid.org/0009-0005-8126-2356Xiaofan Wuhttps://orcid.org/0000-0003-4265-4240 BackgroundAccurate mortality risk prediction is crucial for effective cardiovascular risk management. Recent advancements in artificial intelligence (AI) have demonstrated potential in this specific medical field. Qwen-2 and Llama-3 are high-performance, open-source large language models (LLMs) available online. An artificial neural network (ANN) algorithm derived from the SWEDEHEART (Swedish Web System for Enhancement and Development of Evidence-Based Care in Heart Disease Evaluated According to Recommended Therapies) registry, termed SWEDEHEART-AI, can predict patient prognosis following acute myocardial infarction (AMI). ObjectiveThis study aims to evaluate the 3 models mentioned above in predicting 1-year all-cause mortality in critically ill patients with AMI. MethodsThe Medical Information Mart for Intensive Care IV (MIMIC-IV) database is a publicly available data set in critical care medicine. We included 2758 patients who were first admitted for AMI and discharged alive. SWEDEHEART-AI calculated the mortality rate based on each patient’s 21 clinical variables. Qwen-2 and Llama-3 analyzed the content of patients’ discharge records and directly provided a 1-decimal value between 0 and 1 to represent 1-year death risk probabilities. The patients’ actual mortality was verified using follow-up data. The predictive performance of the 3 models was assessed and compared using the Harrell C-statistic (C-index), the area under the receiver operating characteristic curve (AUROC), calibration plots, Kaplan-Meier curves, and decision curve analysis. ResultsSWEDEHEART-AI demonstrated strong discrimination in predicting 1-year all-cause mortality in patients with AMI, with a higher C-index than Qwen-2 and Llama-3 (C-index 0.72, 95% CI 0.69-0.74 vs C-index 0.65, 0.62-0.67 vs C-index 0.56, 95% CI 0.53-0.58, respectively; all P<.001 for both comparisons). SWEDEHEART-AI also showed high and consistent AUROC in the time-dependent ROC curve. The death rates calculated by SWEDEHEART-AI were positively correlated with actual mortality, and the 3 risk classes derived from this model showed clear differentiation in the Kaplan-Meier curve (P<.001). Calibration plots indicated that SWEDEHEART-AI tended to overestimate mortality risk, with an observed-to-expected ratio of 0.478. Compared with the LLMs, SWEDEHEART-AI demonstrated positive and greater net benefits at risk thresholds below 19%. ConclusionsSWEDEHEART-AI, a trained ANN model, demonstrated the best performance, with strong discrimination and clinical utility in predicting 1-year all-cause mortality in patients with AMI from an intensive care cohort. Among the LLMs, Qwen-2 outperformed Llama-3 and showed moderate predictive value. Qwen-2 and SWEDEHEART-AI exhibited comparable classification effectiveness. The future integration of LLMs into clinical decision support systems holds promise for accurate risk stratification in patients with AMI; however, further research is needed to optimize LLM performance and address calibration issues across diverse patient populations.https://www.jmir.org/2025/1/e67253 |
| spellingShingle | Boqun Shi Liangguo Chen Shuo Pang Yue Wang Shen Wang Fadong Li Wenxin Zhao Pengrong Guo Leli Zhang Chu Fan Yi Zou Xiaofan Wu Large Language Models and Artificial Neural Networks for Assessing 1-Year Mortality in Patients With Myocardial Infarction: Analysis From the Medical Information Mart for Intensive Care IV (MIMIC-IV) Database Journal of Medical Internet Research |
| title | Large Language Models and Artificial Neural Networks for Assessing 1-Year Mortality in Patients With Myocardial Infarction: Analysis From the Medical Information Mart for Intensive Care IV (MIMIC-IV) Database |
| title_full | Large Language Models and Artificial Neural Networks for Assessing 1-Year Mortality in Patients With Myocardial Infarction: Analysis From the Medical Information Mart for Intensive Care IV (MIMIC-IV) Database |
| title_fullStr | Large Language Models and Artificial Neural Networks for Assessing 1-Year Mortality in Patients With Myocardial Infarction: Analysis From the Medical Information Mart for Intensive Care IV (MIMIC-IV) Database |
| title_full_unstemmed | Large Language Models and Artificial Neural Networks for Assessing 1-Year Mortality in Patients With Myocardial Infarction: Analysis From the Medical Information Mart for Intensive Care IV (MIMIC-IV) Database |
| title_short | Large Language Models and Artificial Neural Networks for Assessing 1-Year Mortality in Patients With Myocardial Infarction: Analysis From the Medical Information Mart for Intensive Care IV (MIMIC-IV) Database |
| title_sort | large language models and artificial neural networks for assessing 1 year mortality in patients with myocardial infarction analysis from the medical information mart for intensive care iv mimic iv database |
| url | https://www.jmir.org/2025/1/e67253 |
| work_keys_str_mv | AT boqunshi largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT liangguochen largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT shuopang largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT yuewang largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT shenwang largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT fadongli largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT wenxinzhao largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT pengrongguo largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT lelizhang largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT chufan largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT yizou largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase AT xiaofanwu largelanguagemodelsandartificialneuralnetworksforassessing1yearmortalityinpatientswithmyocardialinfarctionanalysisfromthemedicalinformationmartforintensivecareivmimicivdatabase |