A multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on CT angiography
Abstract Purpose To validate JLK-LVO, a software detecting large vessel occlusion (LVO) on computed tomography angiography (CTA), within a multicenter dataset. Methods From 2021 to 2023, we enrolled patients with ischemic stroke who underwent CTA within 24-hour of onset at six university hospitals f...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-03-01
|
| Series: | BMC Neurology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12883-025-04107-6 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850039360554008576 |
|---|---|
| author | Kyu Sun Yum Jong-Won Chung Sue Young Ha Kwang-Yeol Park Dong-Ick Shin Hong-Kyun Park Yong-Jin Cho Keun-Sik Hong Jae Guk Kim Soo Joo Lee Joon-Tae Kim Woo-Keun Seo Oh Young Bang Gyeong-Moon Kim Myungjae Lee Dongmin Kim Leonard Sunwoo Hee-Joon Bae Wi-Sun Ryu Beom Joon Kim |
| author_facet | Kyu Sun Yum Jong-Won Chung Sue Young Ha Kwang-Yeol Park Dong-Ick Shin Hong-Kyun Park Yong-Jin Cho Keun-Sik Hong Jae Guk Kim Soo Joo Lee Joon-Tae Kim Woo-Keun Seo Oh Young Bang Gyeong-Moon Kim Myungjae Lee Dongmin Kim Leonard Sunwoo Hee-Joon Bae Wi-Sun Ryu Beom Joon Kim |
| author_sort | Kyu Sun Yum |
| collection | DOAJ |
| description | Abstract Purpose To validate JLK-LVO, a software detecting large vessel occlusion (LVO) on computed tomography angiography (CTA), within a multicenter dataset. Methods From 2021 to 2023, we enrolled patients with ischemic stroke who underwent CTA within 24-hour of onset at six university hospitals for validation and calibration datasets and at another university hospital for an independent dataset for testing model calibration. The diagnostic performance was evaluated using area under the curve (AUC), sensitivity, and specificity across the entire study population and specifically in patients with isolated middle cerebral artery (MCA)-M2 occlusion. We calibrated LVO probabilities using logistic regression and by grouping LVO probabilities based on observed frequency. Results After excluding 168 patients, 796 remained; the mean (SD) age was 68.9 (13.7) years, and 57.7% were men. LVO was present in 193 (24.3%) of patients, and the median interval from last-known-well to CTA was 5.7 h (IQR 2.5–12.1 h). The software achieved an AUC of 0.944 (95% CI 0.926–0.960), with a sensitivity of 89.6% (84.5–93.6%) and a specificity of 90.4% (87.7–92.6%). In isolated MCA-M2 occlusion, the AUROC was 0.880 (95% CI 0.824–0.921). Due to sparse data between 20 and 60% of LVO probabilities, recategorization into unlikely (0–20% LVO scores), less likely (20–60%), possible (60–90%), and suggestive (90–100%) provided a reliable estimation of LVO compared with mathematical calibration. The category of LVO probabilities was associated with follow-up infarct volumes and functional outcome. Conclusion In this multicenter study, we proved the clinical efficacy of the software in detecting LVO on CTA. |
| format | Article |
| id | doaj-art-e114176da0f44ef298ffc97df7e54960 |
| institution | DOAJ |
| issn | 1471-2377 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Neurology |
| spelling | doaj-art-e114176da0f44ef298ffc97df7e549602025-08-20T02:56:21ZengBMCBMC Neurology1471-23772025-03-0125111010.1186/s12883-025-04107-6A multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on CT angiographyKyu Sun Yum0Jong-Won Chung1Sue Young Ha2Kwang-Yeol Park3Dong-Ick Shin4Hong-Kyun Park5Yong-Jin Cho6Keun-Sik Hong7Jae Guk Kim8Soo Joo Lee9Joon-Tae Kim10Woo-Keun Seo11Oh Young Bang12Gyeong-Moon Kim13Myungjae Lee14Dongmin Kim15Leonard Sunwoo16Hee-Joon Bae17Wi-Sun Ryu18Beom Joon Kim19Department of Neurology, College of Medicine, Chungbuk National University Hospital, Chungbuk National UniversityDepartment of Neurology, Samsung Medical Center, Sungkyunkwan University College of MedicineArtificial Intelligence Research Center, JLK IncDepartment of Neurology, Chung-Ang University College of Medicine, Chung-Ang University HospitalDepartment of Neurology, College of Medicine, Chungbuk National University Hospital, Chungbuk National UniversityDepartment of Neurology, Inje University Ilsan Paik Hospital, Inje University College of MedicineDepartment of Neurology, Inje University Ilsan Paik Hospital, Inje University College of MedicineDepartment of Neurology, Inje University Ilsan Paik Hospital, Inje University College of MedicineDepartment of Neurology, Daejeon Eulji Medical Center, Eulji University School of MedicineDepartment of Neurology, Daejeon Eulji Medical Center, Eulji University School of MedicineDepartment of Neurology, Chonnam National University Hospital, Chonnam National University Medical SchoolDepartment of Neurology, Samsung Medical Center, Sungkyunkwan University College of MedicineDepartment of Neurology, Samsung Medical Center, Sungkyunkwan University College of MedicineDepartment of Neurology, Samsung Medical Center, Sungkyunkwan University College of MedicineArtificial Intelligence Research Center, JLK IncArtificial Intelligence Research Center, JLK IncDepartment of Radiology, Seoul National University Bundang Hospital, Seoul National University College of MedicineDepartment of Neurology, Seoul National University College of MedicineArtificial Intelligence Research Center, JLK IncDepartment of Neurology, Seoul National University College of MedicineAbstract Purpose To validate JLK-LVO, a software detecting large vessel occlusion (LVO) on computed tomography angiography (CTA), within a multicenter dataset. Methods From 2021 to 2023, we enrolled patients with ischemic stroke who underwent CTA within 24-hour of onset at six university hospitals for validation and calibration datasets and at another university hospital for an independent dataset for testing model calibration. The diagnostic performance was evaluated using area under the curve (AUC), sensitivity, and specificity across the entire study population and specifically in patients with isolated middle cerebral artery (MCA)-M2 occlusion. We calibrated LVO probabilities using logistic regression and by grouping LVO probabilities based on observed frequency. Results After excluding 168 patients, 796 remained; the mean (SD) age was 68.9 (13.7) years, and 57.7% were men. LVO was present in 193 (24.3%) of patients, and the median interval from last-known-well to CTA was 5.7 h (IQR 2.5–12.1 h). The software achieved an AUC of 0.944 (95% CI 0.926–0.960), with a sensitivity of 89.6% (84.5–93.6%) and a specificity of 90.4% (87.7–92.6%). In isolated MCA-M2 occlusion, the AUROC was 0.880 (95% CI 0.824–0.921). Due to sparse data between 20 and 60% of LVO probabilities, recategorization into unlikely (0–20% LVO scores), less likely (20–60%), possible (60–90%), and suggestive (90–100%) provided a reliable estimation of LVO compared with mathematical calibration. The category of LVO probabilities was associated with follow-up infarct volumes and functional outcome. Conclusion In this multicenter study, we proved the clinical efficacy of the software in detecting LVO on CTA.https://doi.org/10.1186/s12883-025-04107-6Artificial intelligenceComputed tomography angiographyLarge vessel occlusionIschemic stroke |
| spellingShingle | Kyu Sun Yum Jong-Won Chung Sue Young Ha Kwang-Yeol Park Dong-Ick Shin Hong-Kyun Park Yong-Jin Cho Keun-Sik Hong Jae Guk Kim Soo Joo Lee Joon-Tae Kim Woo-Keun Seo Oh Young Bang Gyeong-Moon Kim Myungjae Lee Dongmin Kim Leonard Sunwoo Hee-Joon Bae Wi-Sun Ryu Beom Joon Kim A multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on CT angiography BMC Neurology Artificial intelligence Computed tomography angiography Large vessel occlusion Ischemic stroke |
| title | A multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on CT angiography |
| title_full | A multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on CT angiography |
| title_fullStr | A multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on CT angiography |
| title_full_unstemmed | A multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on CT angiography |
| title_short | A multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on CT angiography |
| title_sort | multicenter validation and calibration of automated software package for detecting anterior circulation large vessel occlusion on ct angiography |
| topic | Artificial intelligence Computed tomography angiography Large vessel occlusion Ischemic stroke |
| url | https://doi.org/10.1186/s12883-025-04107-6 |
| work_keys_str_mv | AT kyusunyum amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT jongwonchung amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT sueyoungha amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT kwangyeolpark amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT dongickshin amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT hongkyunpark amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT yongjincho amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT keunsikhong amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT jaegukkim amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT soojoolee amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT joontaekim amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT wookeunseo amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT ohyoungbang amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT gyeongmoonkim amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT myungjaelee amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT dongminkim amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT leonardsunwoo amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT heejoonbae amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT wisunryu amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT beomjoonkim amulticentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT kyusunyum multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT jongwonchung multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT sueyoungha multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT kwangyeolpark multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT dongickshin multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT hongkyunpark multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT yongjincho multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT keunsikhong multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT jaegukkim multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT soojoolee multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT joontaekim multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT wookeunseo multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT ohyoungbang multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT gyeongmoonkim multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT myungjaelee multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT dongminkim multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT leonardsunwoo multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT heejoonbae multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT wisunryu multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography AT beomjoonkim multicentervalidationandcalibrationofautomatedsoftwarepackagefordetectinganteriorcirculationlargevesselocclusiononctangiography |