A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality.
<h4>Background</h4>An accurate prognostic tool is essential to aid clinical decision-making (e.g., patient triage) and to advance personalized medicine. However, such a prognostic tool is lacking for acute pancreatitis (AP). Increasingly machine learning (ML) techniques are being used to...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2025-02-01
|
| Series: | PLoS Medicine |
| Online Access: | https://doi.org/10.1371/journal.pmed.1004432 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850190272797868032 |
|---|---|
| author | Brian Critelli Amier Hassan Ila Lahooti Lydia Noh Jun Sung Park Kathleen Tong Ali Lahooti Nathan Matzko Jan Niklas Adams Lukas Liss Justin Quion David Restrepo Melica Nikahd Stacey Culp Adam Lacy-Hulbert Cate Speake James Buxbaum Jason Bischof Cemal Yazici Anna Evans-Phillips Sophie Terp Alexandra Weissman Darwin Conwell Philip Hart Mitchell Ramsey Somashekar Krishna Samuel Han Erica Park Raj Shah Venkata Akshintala John A Windsor Nikhil K Mull Georgios Papachristou Leo Anthony Celi Peter Lee |
| author_facet | Brian Critelli Amier Hassan Ila Lahooti Lydia Noh Jun Sung Park Kathleen Tong Ali Lahooti Nathan Matzko Jan Niklas Adams Lukas Liss Justin Quion David Restrepo Melica Nikahd Stacey Culp Adam Lacy-Hulbert Cate Speake James Buxbaum Jason Bischof Cemal Yazici Anna Evans-Phillips Sophie Terp Alexandra Weissman Darwin Conwell Philip Hart Mitchell Ramsey Somashekar Krishna Samuel Han Erica Park Raj Shah Venkata Akshintala John A Windsor Nikhil K Mull Georgios Papachristou Leo Anthony Celi Peter Lee |
| author_sort | Brian Critelli |
| collection | DOAJ |
| description | <h4>Background</h4>An accurate prognostic tool is essential to aid clinical decision-making (e.g., patient triage) and to advance personalized medicine. However, such a prognostic tool is lacking for acute pancreatitis (AP). Increasingly machine learning (ML) techniques are being used to develop high-performing prognostic models in AP. However, methodologic and reporting quality has received little attention. High-quality reporting and study methodology are critical for model validity, reproducibility, and clinical implementation. In collaboration with content experts in ML methodology, we performed a systematic review critically appraising the quality of methodology and reporting of recently published ML AP prognostic models.<h4>Methods/findings</h4>Using a validated search strategy, we identified ML AP studies from the databases MEDLINE and EMBASE published between January 2021 and December 2023. We also searched pre-print servers medRxiv, bioRxiv, and arXiv for pre-prints registered between January 2021 and December 2023. Eligibility criteria included all retrospective or prospective studies that developed or validated new or existing ML models in patients with AP that predicted an outcome following an episode of AP. Meta-analysis was considered if there was homogeneity in the study design and in the type of outcome predicted. For risk of bias (ROB) assessment, we used the Prediction Model Risk of Bias Assessment Tool. Quality of reporting was assessed using the Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis-Artificial Intelligence (TRIPOD+AI) statement that defines standards for 27 items that should be reported in publications using ML prognostic models. The search strategy identified 6,480 publications of which 30 met the eligibility criteria. Studies originated from China (22), the United States (4), and other (4). All 30 studies developed a new ML model and none sought to validate an existing ML model, producing a total of 39 new ML models. AP severity (23/39) or mortality (6/39) were the most common outcomes predicted. The mean area under the curve for all models and endpoints was 0.91 (SD 0.08). The ROB was high for at least one domain in all 39 models, particularly for the analysis domain (37/39 models). Steps were not taken to minimize over-optimistic model performance in 27/39 models. Due to heterogeneity in the study design and in how the outcomes were defined and determined, meta-analysis was not performed. Studies reported on only 15/27 items from TRIPOD+AI standards, with only 7/30 justifying sample size and 13/30 assessing data quality. Other reporting deficiencies included omissions regarding human-AI interaction (28/30), handling low-quality or incomplete data in practice (27/30), sharing analytical codes (25/30), study protocols (25/30), and reporting source data (19/30).<h4>Conclusions</h4>There are significant deficiencies in the methodology and reporting of recently published ML based prognostic models in AP patients. These undermine the validity, reproducibility, and implementation of these prognostic models despite their promise of superior predictive accuracy.<h4>Registration</h4>Research Registry (reviewregistry1727). |
| format | Article |
| id | doaj-art-e7f8f807254447b4876d4ac12555c939 |
| institution | OA Journals |
| issn | 1549-1277 1549-1676 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Medicine |
| spelling | doaj-art-e7f8f807254447b4876d4ac12555c9392025-08-20T02:15:20ZengPublic Library of Science (PLoS)PLoS Medicine1549-12771549-16762025-02-01222e100443210.1371/journal.pmed.1004432A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality.Brian CritelliAmier HassanIla LahootiLydia NohJun Sung ParkKathleen TongAli LahootiNathan MatzkoJan Niklas AdamsLukas LissJustin QuionDavid RestrepoMelica NikahdStacey CulpAdam Lacy-HulbertCate SpeakeJames BuxbaumJason BischofCemal YaziciAnna Evans-PhillipsSophie TerpAlexandra WeissmanDarwin ConwellPhilip HartMitchell RamseySomashekar KrishnaSamuel HanErica ParkRaj ShahVenkata AkshintalaJohn A WindsorNikhil K MullGeorgios PapachristouLeo Anthony CeliPeter Lee<h4>Background</h4>An accurate prognostic tool is essential to aid clinical decision-making (e.g., patient triage) and to advance personalized medicine. However, such a prognostic tool is lacking for acute pancreatitis (AP). Increasingly machine learning (ML) techniques are being used to develop high-performing prognostic models in AP. However, methodologic and reporting quality has received little attention. High-quality reporting and study methodology are critical for model validity, reproducibility, and clinical implementation. In collaboration with content experts in ML methodology, we performed a systematic review critically appraising the quality of methodology and reporting of recently published ML AP prognostic models.<h4>Methods/findings</h4>Using a validated search strategy, we identified ML AP studies from the databases MEDLINE and EMBASE published between January 2021 and December 2023. We also searched pre-print servers medRxiv, bioRxiv, and arXiv for pre-prints registered between January 2021 and December 2023. Eligibility criteria included all retrospective or prospective studies that developed or validated new or existing ML models in patients with AP that predicted an outcome following an episode of AP. Meta-analysis was considered if there was homogeneity in the study design and in the type of outcome predicted. For risk of bias (ROB) assessment, we used the Prediction Model Risk of Bias Assessment Tool. Quality of reporting was assessed using the Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis-Artificial Intelligence (TRIPOD+AI) statement that defines standards for 27 items that should be reported in publications using ML prognostic models. The search strategy identified 6,480 publications of which 30 met the eligibility criteria. Studies originated from China (22), the United States (4), and other (4). All 30 studies developed a new ML model and none sought to validate an existing ML model, producing a total of 39 new ML models. AP severity (23/39) or mortality (6/39) were the most common outcomes predicted. The mean area under the curve for all models and endpoints was 0.91 (SD 0.08). The ROB was high for at least one domain in all 39 models, particularly for the analysis domain (37/39 models). Steps were not taken to minimize over-optimistic model performance in 27/39 models. Due to heterogeneity in the study design and in how the outcomes were defined and determined, meta-analysis was not performed. Studies reported on only 15/27 items from TRIPOD+AI standards, with only 7/30 justifying sample size and 13/30 assessing data quality. Other reporting deficiencies included omissions regarding human-AI interaction (28/30), handling low-quality or incomplete data in practice (27/30), sharing analytical codes (25/30), study protocols (25/30), and reporting source data (19/30).<h4>Conclusions</h4>There are significant deficiencies in the methodology and reporting of recently published ML based prognostic models in AP patients. These undermine the validity, reproducibility, and implementation of these prognostic models despite their promise of superior predictive accuracy.<h4>Registration</h4>Research Registry (reviewregistry1727).https://doi.org/10.1371/journal.pmed.1004432 |
| spellingShingle | Brian Critelli Amier Hassan Ila Lahooti Lydia Noh Jun Sung Park Kathleen Tong Ali Lahooti Nathan Matzko Jan Niklas Adams Lukas Liss Justin Quion David Restrepo Melica Nikahd Stacey Culp Adam Lacy-Hulbert Cate Speake James Buxbaum Jason Bischof Cemal Yazici Anna Evans-Phillips Sophie Terp Alexandra Weissman Darwin Conwell Philip Hart Mitchell Ramsey Somashekar Krishna Samuel Han Erica Park Raj Shah Venkata Akshintala John A Windsor Nikhil K Mull Georgios Papachristou Leo Anthony Celi Peter Lee A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality. PLoS Medicine |
| title | A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality. |
| title_full | A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality. |
| title_fullStr | A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality. |
| title_full_unstemmed | A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality. |
| title_short | A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality. |
| title_sort | systematic review of machine learning based prognostic models for acute pancreatitis towards improving methods and reporting quality |
| url | https://doi.org/10.1371/journal.pmed.1004432 |
| work_keys_str_mv | AT briancritelli asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT amierhassan asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT ilalahooti asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT lydianoh asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT junsungpark asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT kathleentong asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT alilahooti asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT nathanmatzko asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT janniklasadams asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT lukasliss asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT justinquion asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT davidrestrepo asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT melicanikahd asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT staceyculp asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT adamlacyhulbert asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT catespeake asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT jamesbuxbaum asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT jasonbischof asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT cemalyazici asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT annaevansphillips asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT sophieterp asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT alexandraweissman asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT darwinconwell asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT philiphart asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT mitchellramsey asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT somashekarkrishna asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT samuelhan asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT ericapark asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT rajshah asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT venkataakshintala asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT johnawindsor asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT nikhilkmull asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT georgiospapachristou asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT leoanthonyceli asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT peterlee asystematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT briancritelli systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT amierhassan systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT ilalahooti systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT lydianoh systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT junsungpark systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT kathleentong systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT alilahooti systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT nathanmatzko systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT janniklasadams systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT lukasliss systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT justinquion systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT davidrestrepo systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT melicanikahd systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT staceyculp systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT adamlacyhulbert systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT catespeake systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT jamesbuxbaum systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT jasonbischof systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT cemalyazici systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT annaevansphillips systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT sophieterp systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT alexandraweissman systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT darwinconwell systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT philiphart systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT mitchellramsey systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT somashekarkrishna systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT samuelhan systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT ericapark systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT rajshah systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT venkataakshintala systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT johnawindsor systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT nikhilkmull systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT georgiospapachristou systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT leoanthonyceli systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality AT peterlee systematicreviewofmachinelearningbasedprognosticmodelsforacutepancreatitistowardsimprovingmethodsandreportingquality |