Expert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unit
Abstract Background Determining extubation readiness in pediatric intensive care units (PICU) is challenging. We used expert-augmented machine learning (EAML), a method that combines machine learning with human expert knowledge, to predict successful extubation. Methods We extracted electronic healt...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-07-01
|
| Series: | BMC Medical Informatics and Decision Making |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12911-025-03070-z |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849334547636813824 |
|---|---|
| author | Jean Digitale Deborah Franzon Jin Ge Charles McCulloch Mark J. Pletcher Efstathios D. Gennatas |
| author_facet | Jean Digitale Deborah Franzon Jin Ge Charles McCulloch Mark J. Pletcher Efstathios D. Gennatas |
| author_sort | Jean Digitale |
| collection | DOAJ |
| description | Abstract Background Determining extubation readiness in pediatric intensive care units (PICU) is challenging. We used expert-augmented machine learning (EAML), a method that combines machine learning with human expert knowledge, to predict successful extubation. Methods We extracted electronic health record data from patients in two PICUs. Data from patients in one unit was split into 80% training and 20% test, while patients in the other served as an external test set. EAML begins by training RuleFit, which converts gradient-boosted trees into decision rules. Then, expert clinicians were asked to assess the relative probability of successful extubation of the subgroup defined by each rule compared with the entire sample. The rules were ranked in order of increasing chance of successful extubation according to (1) the RuleFit model and (2) clinician assessment, and differences between the two rankings were calculated. The initial RuleFit model was then regularized based on these differences, producing the EAML model. Results The RuleFit model selected 46 rules; we surveyed 25 clinician experts to provide feedback on them. All clinicians worked in a PICU setting and were from multiple disciplines; over half (56%) had > 5 years of PICU experience. As expected, the added regularization slightly lowered performance of EAML compared with RuleFit in the internal test set, although the difference was not statistically significant (RuleFit AUC = 0.817 vs. best-performing EAML model AUC = 0.814, difference = 0.003, 95% CI of difference = -0.009, 0.003). EAML had superior performance in the external test set (RuleFit AUC = 0.791 vs. best-performing EAML model AUC = 0.799, difference = 0.007, 95% CI of difference = 0.002, 0.013). Conclusions When creating a model to predict successful extubation in PICU patients, incorporating expert knowledge directly into the model construction process via EAML produced a model more generalizable to an external test set. |
| format | Article |
| id | doaj-art-803cb39f34674c3fb8014fcbd0a38e77 |
| institution | Kabale University |
| issn | 1472-6947 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Informatics and Decision Making |
| spelling | doaj-art-803cb39f34674c3fb8014fcbd0a38e772025-08-20T03:45:32ZengBMCBMC Medical Informatics and Decision Making1472-69472025-07-0125111110.1186/s12911-025-03070-zExpert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unitJean Digitale0Deborah Franzon1Jin Ge2Charles McCulloch3Mark J. Pletcher4Efstathios D. Gennatas5Department of Epidemiology and Biostatistics, University of California, San FranciscoDepartment of Pediatrics, Benioff Children’s Hospital, University of California, San FranciscoDivision of Gastroenterology and Hepatology, Department of Medicine, University of CaliforniaDepartment of Epidemiology and Biostatistics, University of California, San FranciscoDepartment of Epidemiology and Biostatistics, University of California, San FranciscoDepartment of Epidemiology and Biostatistics, University of California, San FranciscoAbstract Background Determining extubation readiness in pediatric intensive care units (PICU) is challenging. We used expert-augmented machine learning (EAML), a method that combines machine learning with human expert knowledge, to predict successful extubation. Methods We extracted electronic health record data from patients in two PICUs. Data from patients in one unit was split into 80% training and 20% test, while patients in the other served as an external test set. EAML begins by training RuleFit, which converts gradient-boosted trees into decision rules. Then, expert clinicians were asked to assess the relative probability of successful extubation of the subgroup defined by each rule compared with the entire sample. The rules were ranked in order of increasing chance of successful extubation according to (1) the RuleFit model and (2) clinician assessment, and differences between the two rankings were calculated. The initial RuleFit model was then regularized based on these differences, producing the EAML model. Results The RuleFit model selected 46 rules; we surveyed 25 clinician experts to provide feedback on them. All clinicians worked in a PICU setting and were from multiple disciplines; over half (56%) had > 5 years of PICU experience. As expected, the added regularization slightly lowered performance of EAML compared with RuleFit in the internal test set, although the difference was not statistically significant (RuleFit AUC = 0.817 vs. best-performing EAML model AUC = 0.814, difference = 0.003, 95% CI of difference = -0.009, 0.003). EAML had superior performance in the external test set (RuleFit AUC = 0.791 vs. best-performing EAML model AUC = 0.799, difference = 0.007, 95% CI of difference = 0.002, 0.013). Conclusions When creating a model to predict successful extubation in PICU patients, incorporating expert knowledge directly into the model construction process via EAML produced a model more generalizable to an external test set.https://doi.org/10.1186/s12911-025-03070-zExpert-augmented machine learningClinical prediction modelMachine learningInterpretability |
| spellingShingle | Jean Digitale Deborah Franzon Jin Ge Charles McCulloch Mark J. Pletcher Efstathios D. Gennatas Expert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unit BMC Medical Informatics and Decision Making Expert-augmented machine learning Clinical prediction model Machine learning Interpretability |
| title | Expert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unit |
| title_full | Expert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unit |
| title_fullStr | Expert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unit |
| title_full_unstemmed | Expert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unit |
| title_short | Expert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unit |
| title_sort | expert augmented machine learning for predicting extubation readiness in the pediatric intensive care unit |
| topic | Expert-augmented machine learning Clinical prediction model Machine learning Interpretability |
| url | https://doi.org/10.1186/s12911-025-03070-z |
| work_keys_str_mv | AT jeandigitale expertaugmentedmachinelearningforpredictingextubationreadinessinthepediatricintensivecareunit AT deborahfranzon expertaugmentedmachinelearningforpredictingextubationreadinessinthepediatricintensivecareunit AT jinge expertaugmentedmachinelearningforpredictingextubationreadinessinthepediatricintensivecareunit AT charlesmcculloch expertaugmentedmachinelearningforpredictingextubationreadinessinthepediatricintensivecareunit AT markjpletcher expertaugmentedmachinelearningforpredictingextubationreadinessinthepediatricintensivecareunit AT efstathiosdgennatas expertaugmentedmachinelearningforpredictingextubationreadinessinthepediatricintensivecareunit |