Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model?
In production settings, deep learning models often rely on fixed thresholds. This study investigates whether using varying thresholds over time enhances predictive accuracy and clinical utility, especially for early sepsis prediction. We retrospectively analyzed EMR data from Hallym University Chun...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Iuliu Hatieganu University of Medicine and Pharmacy, Cluj-Napoca
2024-11-01
|
| Series: | Applied Medical Informatics |
| Subjects: | |
| Online Access: | https://ami.info.umfcluj.ro/index.php/AMI/article/view/1073 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841558928476864512 |
|---|---|
| author | Sungsoo HONG Hyunwoo CHOO Kyung Hyun LEE Sungjun HONG Ki-Byung LEE Chang Youl LEE |
| author_facet | Sungsoo HONG Hyunwoo CHOO Kyung Hyun LEE Sungjun HONG Ki-Byung LEE Chang Youl LEE |
| author_sort | Sungsoo HONG |
| collection | DOAJ |
| description |
In production settings, deep learning models often rely on fixed thresholds. This study investigates whether using varying thresholds over time enhances predictive accuracy and clinical utility, especially for early sepsis prediction. We retrospectively analyzed EMR data from Hallym University Chuncheon Sacred Heart Hospital (2018-2022), excluding patients aged under 18 or without vital signs. Utilizing the AITRICS-VC SEPS deep learning model, which predicts sepsis using six vital signs, eleven lab results and patient information, we examined prediction thresholds at one-hour intervals before sepsis onset. Optimal thresholds for each interval were identified using the Youden index. Net benefit and decision curve analysis compared the performance of time-varying versus global thresholds. Results show interval-specific thresholds yield higher net benefits and increased true positive detections: 456 (0-1 hour), 122 (1-2 hours), 41 (2-3 hours), and 29 (3-4 hours) before sepsis onset. This suggests dynamically adjusting thresholds over time can improve early sepsis detection and patient outcomes.
|
| format | Article |
| id | doaj-art-8e2ac216f7b742dbaa13d003479bd87c |
| institution | Kabale University |
| issn | 2067-7855 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | Iuliu Hatieganu University of Medicine and Pharmacy, Cluj-Napoca |
| record_format | Article |
| series | Applied Medical Informatics |
| spelling | doaj-art-8e2ac216f7b742dbaa13d003479bd87c2025-01-05T21:07:49ZengIuliu Hatieganu University of Medicine and Pharmacy, Cluj-NapocaApplied Medical Informatics2067-78552024-11-0146Suppl. 2Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model?Sungsoo HONG0Hyunwoo CHOO1Kyung Hyun LEE2Sungjun HONG3Ki-Byung LEE4Chang Youl LEE5AITRICS, Inc., 218 Teheran-ro, Gangnam-gu, 06221 Seoul, Republic of KoreaAITRICS, Inc., 218 Teheran-ro, Gangnam-gu, 06221 Seoul, Republic of KoreaAITRICS, Inc., 218 Teheran-ro, Gangnam-gu, 06221 Seoul, Republic of KoreaMedical AI Research Center, Research Institute for Future Medicine, Samsung Medical Center, 81 Irwon-ro, Gangnam-gu, 06351 Seoul, Republic of KoreaDivision of Pulmonary, Allergy and Critical Care Medicine, Hallym University Chuncheon Sacred Heart Hospital, 77 Sakju-ro, 24253 Chuncheon, Republic of KoreaDivision of Pulmonary, Allergy and Critical Care Medicine, Hallym University Chuncheon Sacred Heart Hospital, 77 Sakju-ro, 24253 Chuncheon, Republic of Korea In production settings, deep learning models often rely on fixed thresholds. This study investigates whether using varying thresholds over time enhances predictive accuracy and clinical utility, especially for early sepsis prediction. We retrospectively analyzed EMR data from Hallym University Chuncheon Sacred Heart Hospital (2018-2022), excluding patients aged under 18 or without vital signs. Utilizing the AITRICS-VC SEPS deep learning model, which predicts sepsis using six vital signs, eleven lab results and patient information, we examined prediction thresholds at one-hour intervals before sepsis onset. Optimal thresholds for each interval were identified using the Youden index. Net benefit and decision curve analysis compared the performance of time-varying versus global thresholds. Results show interval-specific thresholds yield higher net benefits and increased true positive detections: 456 (0-1 hour), 122 (1-2 hours), 41 (2-3 hours), and 29 (3-4 hours) before sepsis onset. This suggests dynamically adjusting thresholds over time can improve early sepsis detection and patient outcomes. https://ami.info.umfcluj.ro/index.php/AMI/article/view/1073Deep learningEarly predictionSepsisThreshold adjustmentNet benefit |
| spellingShingle | Sungsoo HONG Hyunwoo CHOO Kyung Hyun LEE Sungjun HONG Ki-Byung LEE Chang Youl LEE Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model? Applied Medical Informatics Deep learning Early prediction Sepsis Threshold adjustment Net benefit |
| title | Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model? |
| title_full | Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model? |
| title_fullStr | Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model? |
| title_full_unstemmed | Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model? |
| title_short | Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model? |
| title_sort | is it beneficial to use different thresholds over time for early prediction model |
| topic | Deep learning Early prediction Sepsis Threshold adjustment Net benefit |
| url | https://ami.info.umfcluj.ro/index.php/AMI/article/view/1073 |
| work_keys_str_mv | AT sungsoohong isitbeneficialtousedifferentthresholdsovertimeforearlypredictionmodel AT hyunwoochoo isitbeneficialtousedifferentthresholdsovertimeforearlypredictionmodel AT kyunghyunlee isitbeneficialtousedifferentthresholdsovertimeforearlypredictionmodel AT sungjunhong isitbeneficialtousedifferentthresholdsovertimeforearlypredictionmodel AT kibyunglee isitbeneficialtousedifferentthresholdsovertimeforearlypredictionmodel AT changyoullee isitbeneficialtousedifferentthresholdsovertimeforearlypredictionmodel |