Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study
Objective Our study compares physician judgement with an automated early warning system (EWS) for predicting clinical deterioration of hospitalised general internal medicine patients.Design Prospective observational study of clinical predictions made at the end of the daytime work-shift for an acade...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMJ Publishing Group
2019-10-01
|
| Series: | BMJ Open |
| Online Access: | https://bmjopen.bmj.com/content/9/10/e032187.full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850060676095016960 |
|---|---|
| author | Jonathan Arnold Alex Davis Baruch Fischhoff Emmanuelle Yecies Jon Grace Andrew Klobuka Deepika Mohan Janel Hanmer |
| author_facet | Jonathan Arnold Alex Davis Baruch Fischhoff Emmanuelle Yecies Jon Grace Andrew Klobuka Deepika Mohan Janel Hanmer |
| author_sort | Jonathan Arnold |
| collection | DOAJ |
| description | Objective Our study compares physician judgement with an automated early warning system (EWS) for predicting clinical deterioration of hospitalised general internal medicine patients.Design Prospective observational study of clinical predictions made at the end of the daytime work-shift for an academic general internal medicine floor team compared with the risk assessment from an automated EWS collected at the same time.Setting Internal medicine teaching wards at a single tertiary care academic medical centre in the USA.Participants Intern physicians working on the internal medicine wards and an automated EWS (Rothman Index by PeraHealth).Outcome Clinical deterioration within 24 hours including cardiac or pulmonary arrest, rapid response team activation or unscheduled intensive care unit transfer.Results We collected predictions for 1874 patient days and saw 35 clinical deteriorations (1.9%). The area under the receiver operating curve (AUROC) for the EWS was 0.73 vs 0.70 for physicians (p=0.571). A linear regression model combining physician and EWS predictions had an AUROC of 0.75, outperforming physicians (p=0.016) and the EWS (p=0.05).Conclusions There is no significant difference in the performance of the EWS and physicians in predicting clinical deterioration at 24 hours on an inpatient general medicine ward. A combined model outperformed either alone. The EWS and physicians identify partially overlapping sets of at-risk patients suggesting they rely on different cues or decision rules for their predictions.Trial registration number NCT02648828. |
| format | Article |
| id | doaj-art-891d025e967f48879f4f89d3572cbc4c |
| institution | DOAJ |
| issn | 2044-6055 |
| language | English |
| publishDate | 2019-10-01 |
| publisher | BMJ Publishing Group |
| record_format | Article |
| series | BMJ Open |
| spelling | doaj-art-891d025e967f48879f4f89d3572cbc4c2025-08-20T02:50:29ZengBMJ Publishing GroupBMJ Open2044-60552019-10-0191010.1136/bmjopen-2019-032187Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational studyJonathan Arnold0Alex Davis1Baruch Fischhoff2Emmanuelle Yecies3Jon Grace4Andrew Klobuka5Deepika Mohan6Janel Hanmer71 Division of General Internal Medicine, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania, USA2 Engineering and Public Policy, Carnegie Mellon University College of Engineering, Pittsburgh, Pennsylvania, USA2 Engineering and Public Policy, Carnegie Mellon University College of Engineering, Pittsburgh, Pennsylvania, USA1 Division of General Internal Medicine, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania, USA3 Division of Pulmonary & Critical Care Medicine, University of Michigan Department of Internal Medicine, Ann Arbor, Michigan, USA4 Department of Radiology, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania, USA5 Department of Critical Care Medicine, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania, USA1 Division of General Internal Medicine, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania, USAObjective Our study compares physician judgement with an automated early warning system (EWS) for predicting clinical deterioration of hospitalised general internal medicine patients.Design Prospective observational study of clinical predictions made at the end of the daytime work-shift for an academic general internal medicine floor team compared with the risk assessment from an automated EWS collected at the same time.Setting Internal medicine teaching wards at a single tertiary care academic medical centre in the USA.Participants Intern physicians working on the internal medicine wards and an automated EWS (Rothman Index by PeraHealth).Outcome Clinical deterioration within 24 hours including cardiac or pulmonary arrest, rapid response team activation or unscheduled intensive care unit transfer.Results We collected predictions for 1874 patient days and saw 35 clinical deteriorations (1.9%). The area under the receiver operating curve (AUROC) for the EWS was 0.73 vs 0.70 for physicians (p=0.571). A linear regression model combining physician and EWS predictions had an AUROC of 0.75, outperforming physicians (p=0.016) and the EWS (p=0.05).Conclusions There is no significant difference in the performance of the EWS and physicians in predicting clinical deterioration at 24 hours on an inpatient general medicine ward. A combined model outperformed either alone. The EWS and physicians identify partially overlapping sets of at-risk patients suggesting they rely on different cues or decision rules for their predictions.Trial registration number NCT02648828.https://bmjopen.bmj.com/content/9/10/e032187.full |
| spellingShingle | Jonathan Arnold Alex Davis Baruch Fischhoff Emmanuelle Yecies Jon Grace Andrew Klobuka Deepika Mohan Janel Hanmer Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study BMJ Open |
| title | Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study |
| title_full | Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study |
| title_fullStr | Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study |
| title_full_unstemmed | Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study |
| title_short | Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study |
| title_sort | comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients a prospective observational study |
| url | https://bmjopen.bmj.com/content/9/10/e032187.full |
| work_keys_str_mv | AT jonathanarnold comparingthepredictiveabilityofacommercialartificialintelligenceearlywarningsystemwithphysicianjudgementforclinicaldeteriorationinhospitalisedgeneralinternalmedicinepatientsaprospectiveobservationalstudy AT alexdavis comparingthepredictiveabilityofacommercialartificialintelligenceearlywarningsystemwithphysicianjudgementforclinicaldeteriorationinhospitalisedgeneralinternalmedicinepatientsaprospectiveobservationalstudy AT baruchfischhoff comparingthepredictiveabilityofacommercialartificialintelligenceearlywarningsystemwithphysicianjudgementforclinicaldeteriorationinhospitalisedgeneralinternalmedicinepatientsaprospectiveobservationalstudy AT emmanuelleyecies comparingthepredictiveabilityofacommercialartificialintelligenceearlywarningsystemwithphysicianjudgementforclinicaldeteriorationinhospitalisedgeneralinternalmedicinepatientsaprospectiveobservationalstudy AT jongrace comparingthepredictiveabilityofacommercialartificialintelligenceearlywarningsystemwithphysicianjudgementforclinicaldeteriorationinhospitalisedgeneralinternalmedicinepatientsaprospectiveobservationalstudy AT andrewklobuka comparingthepredictiveabilityofacommercialartificialintelligenceearlywarningsystemwithphysicianjudgementforclinicaldeteriorationinhospitalisedgeneralinternalmedicinepatientsaprospectiveobservationalstudy AT deepikamohan comparingthepredictiveabilityofacommercialartificialintelligenceearlywarningsystemwithphysicianjudgementforclinicaldeteriorationinhospitalisedgeneralinternalmedicinepatientsaprospectiveobservationalstudy AT janelhanmer comparingthepredictiveabilityofacommercialartificialintelligenceearlywarningsystemwithphysicianjudgementforclinicaldeteriorationinhospitalisedgeneralinternalmedicinepatientsaprospectiveobservationalstudy |