A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features

<italic>Goal:</italic> The purpose of this study is to evaluate the importance of cardiorespiratory variables within a Reinforcement Learning (RL) recommendation system aimed at establishing optimal strategies for drug treatment of septic patients in the intensive care unit (ICU). <it...

Full description

Saved in:
Bibliographic Details
Main Authors: Cristian Drudi, Maximiliano Mollura, Li-wei H. Lehman, Riccardo Barbieri
Format: Article
Language:English
Published: IEEE 2024-01-01
Series:IEEE Open Journal of Engineering in Medicine and Biology
Subjects:
Online Access:https://ieeexplore.ieee.org/document/10439998/
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1832582299742371840
author Cristian Drudi
Maximiliano Mollura
Li-wei H. Lehman
Riccardo Barbieri
author_facet Cristian Drudi
Maximiliano Mollura
Li-wei H. Lehman
Riccardo Barbieri
author_sort Cristian Drudi
collection DOAJ
description <italic>Goal:</italic> The purpose of this study is to evaluate the importance of cardiorespiratory variables within a Reinforcement Learning (RL) recommendation system aimed at establishing optimal strategies for drug treatment of septic patients in the intensive care unit (ICU). <italic>Methods:</italic> We developed a RL model in order to establish drug administration strategies for septic patients using only a set of cardiorespiratory variables. We then compared this model with other RL models trained with a different set of features. We selected patients meeting the Sepsis-3 criteria from the Multi-parameter Intelligent Monitoring in Intensive Care (MIMIC III) database, resulting in a total of 20,496 ICU admissions. A Markov Decision Process (MDP) was built on the extracted discrete time-series. A policy iteration algorithm was used to obtain the optimal AI policy for the MDP. The policy performance was then evaluated using the WIS estimator. The process was repeated for each set of variables and compared to a set of baseline benchmark policies. <italic>Results:</italic> The model trained with cardiorespiratory variables outperformed all other models considered, resulting in a 95% confidence lower bound score of 97.48. This finding highlights the importance of cardiovascular variables in the clinical RL recommendation system. <italic>Conclusions:</italic> We established an efficient RL model for sepsis treatment in the ICU and demonstrated that cardiorespiratory variables provides critical information in devising optimal policies. Given the potentially continuous availability of cardiorespiratory features extracted from bedside physiological waveform monitoring, the proposed framework paves the way for a real time recommendation system for sepsis treatment.
format Article
id doaj-art-4ed15fba9f5c40ecbaf015f5f45254c3
institution Kabale University
issn 2644-1276
language English
publishDate 2024-01-01
publisher IEEE
record_format Article
series IEEE Open Journal of Engineering in Medicine and Biology
spelling doaj-art-4ed15fba9f5c40ecbaf015f5f45254c32025-01-30T00:03:55ZengIEEEIEEE Open Journal of Engineering in Medicine and Biology2644-12762024-01-01580681510.1109/OJEMB.2024.336723610439998A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory FeaturesCristian Drudi0https://orcid.org/0009-0000-1733-8823Maximiliano Mollura1https://orcid.org/0000-0002-2248-8145Li-wei H. Lehman2https://orcid.org/0000-0002-3782-9977Riccardo Barbieri3https://orcid.org/0000-0001-9381-3833Department of Electronics, Informatics and Engineering, Politecnico di Milano, Milano, ItalyDepartment of Electronics, Informatics and Engineering, Politecnico di Milano, Milano, ItalyInstitute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USADepartment of Electronics, Informatics and Engineering, Politecnico di Milano, Milano, Italy<italic>Goal:</italic> The purpose of this study is to evaluate the importance of cardiorespiratory variables within a Reinforcement Learning (RL) recommendation system aimed at establishing optimal strategies for drug treatment of septic patients in the intensive care unit (ICU). <italic>Methods:</italic> We developed a RL model in order to establish drug administration strategies for septic patients using only a set of cardiorespiratory variables. We then compared this model with other RL models trained with a different set of features. We selected patients meeting the Sepsis-3 criteria from the Multi-parameter Intelligent Monitoring in Intensive Care (MIMIC III) database, resulting in a total of 20,496 ICU admissions. A Markov Decision Process (MDP) was built on the extracted discrete time-series. A policy iteration algorithm was used to obtain the optimal AI policy for the MDP. The policy performance was then evaluated using the WIS estimator. The process was repeated for each set of variables and compared to a set of baseline benchmark policies. <italic>Results:</italic> The model trained with cardiorespiratory variables outperformed all other models considered, resulting in a 95% confidence lower bound score of 97.48. This finding highlights the importance of cardiovascular variables in the clinical RL recommendation system. <italic>Conclusions:</italic> We established an efficient RL model for sepsis treatment in the ICU and demonstrated that cardiorespiratory variables provides critical information in devising optimal policies. Given the potentially continuous availability of cardiorespiratory features extracted from bedside physiological waveform monitoring, the proposed framework paves the way for a real time recommendation system for sepsis treatment.https://ieeexplore.ieee.org/document/10439998/Cardiovascular systemdimensionality reductionintensive care unitreinforcement learningsepsis
spellingShingle Cristian Drudi
Maximiliano Mollura
Li-wei H. Lehman
Riccardo Barbieri
A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features
IEEE Open Journal of Engineering in Medicine and Biology
Cardiovascular system
dimensionality reduction
intensive care unit
reinforcement learning
sepsis
title A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features
title_full A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features
title_fullStr A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features
title_full_unstemmed A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features
title_short A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features
title_sort reinforcement learning model for optimal treatment strategies in intensive care assessment of the role of cardiorespiratory features
topic Cardiovascular system
dimensionality reduction
intensive care unit
reinforcement learning
sepsis
url https://ieeexplore.ieee.org/document/10439998/
work_keys_str_mv AT cristiandrudi areinforcementlearningmodelforoptimaltreatmentstrategiesinintensivecareassessmentoftheroleofcardiorespiratoryfeatures
AT maximilianomollura areinforcementlearningmodelforoptimaltreatmentstrategiesinintensivecareassessmentoftheroleofcardiorespiratoryfeatures
AT liweihlehman areinforcementlearningmodelforoptimaltreatmentstrategiesinintensivecareassessmentoftheroleofcardiorespiratoryfeatures
AT riccardobarbieri areinforcementlearningmodelforoptimaltreatmentstrategiesinintensivecareassessmentoftheroleofcardiorespiratoryfeatures
AT cristiandrudi reinforcementlearningmodelforoptimaltreatmentstrategiesinintensivecareassessmentoftheroleofcardiorespiratoryfeatures
AT maximilianomollura reinforcementlearningmodelforoptimaltreatmentstrategiesinintensivecareassessmentoftheroleofcardiorespiratoryfeatures
AT liweihlehman reinforcementlearningmodelforoptimaltreatmentstrategiesinintensivecareassessmentoftheroleofcardiorespiratoryfeatures
AT riccardobarbieri reinforcementlearningmodelforoptimaltreatmentstrategiesinintensivecareassessmentoftheroleofcardiorespiratoryfeatures