A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug Administration
Introduction: Medical device recalls are important to the practice of emergency medicine, as unsafe devices include many ubiquitous items in emergency care, such as vascular access devices, ventilators, infusion pumps, video laryngoscopes, pulse oximetry sensors, and implantable cardioverter defibri...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
eScholarship Publishing, University of California
2024-11-01
|
| Series: | Western Journal of Emergency Medicine |
| Online Access: | https://escholarship.org/uc/item/4sd03611 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1825201882677641216 |
|---|---|
| author | Victor Barbosa Slivinskis Isabela Agi Maluli Joshua Seth Broder |
| author_facet | Victor Barbosa Slivinskis Isabela Agi Maluli Joshua Seth Broder |
| author_sort | Victor Barbosa Slivinskis |
| collection | DOAJ |
| description | Introduction: Medical device recalls are important to the practice of emergency medicine, as unsafe devices include many ubiquitous items in emergency care, such as vascular access devices, ventilators, infusion pumps, video laryngoscopes, pulse oximetry sensors, and implantable cardioverter defibrillators. Identification of dangerous medical devices as early as possible is necessary to minimize patient harms while avoiding false positives to prevent removal of safe devices from use. While the United States Food and Drug Administration (FDA) employs an adverse event reporting program (MedWatch) and database (MAUDE), other data sources and methods might have utility to identify potentially dangerous medical devices. Our objective was to evaluate the sensitivity, specificity, and accuracy of a machine learning (ML) algorithm using publicly available data to predict medical device recalls by the FDA. Methods: We identified recalled medical devices (RMD) and non-recalled medical devices (NRMD) using the FDA’s website and online database. We constructed an ML algorithm (random forest regressor) that automatically searched Google Trends and PubMed for the RMDs and NRMDs. The algorithm was trained using 400 randomly selected devices and then tested using 100 unique random devices. The algorithm output a continuous value (0–1) for recall probability for each device, which were rounded for dichotomous analysis. We determined sensitivity, specificity, and accuracy for each of three time periods prior to recall (T-3, 6, or 12 months), using FDA recall status as the reference standard. The study adhered to relevant items of the Standards for Reporting Diagnostic accuracy studies (STARD) guidelines. Results: Using a rounding threshold of 0.5, sensitivities for T-3, T-6, and T-12 were 89% (95% confidence interval [CI] 69–97), 90% (95% CI 70–97), and 75% (95% CI 53–89). Specificity was 100% (95% CI 95–100) for all three time periods. Accuracy was 98% (95% CI 93–99) for T-3 and T-6, and 95% (95% CI 89–99) for T-12. Using tailored thresholds yielded similar results. Conclusion: An ML algorithm accurately predicted medical device recall status by the FDA with lead times as great as 12 months. Future research could incorporate longer lead times and data sources including FDA reports and prospectively test the ability of ML algorithms to predict FDA recall. |
| format | Article |
| id | doaj-art-1b97d24c1ec34ecc9c41e250999e3011 |
| institution | Kabale University |
| issn | 1936-900X 1936-9018 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | eScholarship Publishing, University of California |
| record_format | Article |
| series | Western Journal of Emergency Medicine |
| spelling | doaj-art-1b97d24c1ec34ecc9c41e250999e30112025-02-07T16:29:04ZengeScholarship Publishing, University of CaliforniaWestern Journal of Emergency Medicine1936-900X1936-90182024-11-0126116117010.5811/westjem.2123821238A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug AdministrationVictor Barbosa Slivinskis0Isabela Agi Maluli1Joshua Seth Broder2Duke University, Pratt School of Engineering, Department of Biomedical Engineering, Durham, North CarolinaDuke University, Trinity College of Arts and Sciences, Department of Chemistry, Durham, North CarolinaDuke University School of Medicine, Department of Emergency Medicine, Durham, North CarolinaIntroduction: Medical device recalls are important to the practice of emergency medicine, as unsafe devices include many ubiquitous items in emergency care, such as vascular access devices, ventilators, infusion pumps, video laryngoscopes, pulse oximetry sensors, and implantable cardioverter defibrillators. Identification of dangerous medical devices as early as possible is necessary to minimize patient harms while avoiding false positives to prevent removal of safe devices from use. While the United States Food and Drug Administration (FDA) employs an adverse event reporting program (MedWatch) and database (MAUDE), other data sources and methods might have utility to identify potentially dangerous medical devices. Our objective was to evaluate the sensitivity, specificity, and accuracy of a machine learning (ML) algorithm using publicly available data to predict medical device recalls by the FDA. Methods: We identified recalled medical devices (RMD) and non-recalled medical devices (NRMD) using the FDA’s website and online database. We constructed an ML algorithm (random forest regressor) that automatically searched Google Trends and PubMed for the RMDs and NRMDs. The algorithm was trained using 400 randomly selected devices and then tested using 100 unique random devices. The algorithm output a continuous value (0–1) for recall probability for each device, which were rounded for dichotomous analysis. We determined sensitivity, specificity, and accuracy for each of three time periods prior to recall (T-3, 6, or 12 months), using FDA recall status as the reference standard. The study adhered to relevant items of the Standards for Reporting Diagnostic accuracy studies (STARD) guidelines. Results: Using a rounding threshold of 0.5, sensitivities for T-3, T-6, and T-12 were 89% (95% confidence interval [CI] 69–97), 90% (95% CI 70–97), and 75% (95% CI 53–89). Specificity was 100% (95% CI 95–100) for all three time periods. Accuracy was 98% (95% CI 93–99) for T-3 and T-6, and 95% (95% CI 89–99) for T-12. Using tailored thresholds yielded similar results. Conclusion: An ML algorithm accurately predicted medical device recall status by the FDA with lead times as great as 12 months. Future research could incorporate longer lead times and data sources including FDA reports and prospectively test the ability of ML algorithms to predict FDA recall.https://escholarship.org/uc/item/4sd03611 |
| spellingShingle | Victor Barbosa Slivinskis Isabela Agi Maluli Joshua Seth Broder A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug Administration Western Journal of Emergency Medicine |
| title | A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug Administration |
| title_full | A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug Administration |
| title_fullStr | A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug Administration |
| title_full_unstemmed | A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug Administration |
| title_short | A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug Administration |
| title_sort | machine learning algorithm to predict medical device recall by the food and drug administration |
| url | https://escholarship.org/uc/item/4sd03611 |
| work_keys_str_mv | AT victorbarbosaslivinskis amachinelearningalgorithmtopredictmedicaldevicerecallbythefoodanddrugadministration AT isabelaagimaluli amachinelearningalgorithmtopredictmedicaldevicerecallbythefoodanddrugadministration AT joshuasethbroder amachinelearningalgorithmtopredictmedicaldevicerecallbythefoodanddrugadministration AT victorbarbosaslivinskis machinelearningalgorithmtopredictmedicaldevicerecallbythefoodanddrugadministration AT isabelaagimaluli machinelearningalgorithmtopredictmedicaldevicerecallbythefoodanddrugadministration AT joshuasethbroder machinelearningalgorithmtopredictmedicaldevicerecallbythefoodanddrugadministration |