Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept Study
Introduction: Timely thrombolytic therapy improves outcomes in acute ischemic stroke. Manual chart review to screen for thrombolysis contraindications may be time-consuming and prone to errors. We developed and tested a large language model (LLM)-based tool to identify thrombolysis contra...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Karger Publishers
2025-03-01
|
| Series: | Cerebrovascular Diseases Extra |
| Online Access: | https://karger.com/article/doi/10.1159/000545317 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849321572981014528 |
|---|---|
| author | Bing Yu Chen Fares Antaki Marco Gonzalez Ken Uchino Samer Albahra Scott Robertson Sidonie Ibrikji Eric Aube Andrew Russman Muhammad Shazam Hussain |
| author_facet | Bing Yu Chen Fares Antaki Marco Gonzalez Ken Uchino Samer Albahra Scott Robertson Sidonie Ibrikji Eric Aube Andrew Russman Muhammad Shazam Hussain |
| author_sort | Bing Yu Chen |
| collection | DOAJ |
| description |
Introduction: Timely thrombolytic therapy improves outcomes in acute ischemic stroke. Manual chart review to screen for thrombolysis contraindications may be time-consuming and prone to errors. We developed and tested a large language model (LLM)-based tool to identify thrombolysis contraindications from clinical notes using synthetic data in a proof-of-concept study. Methods: We generated 150 synthetic clinical notes containing randomly assigned thrombolysis contraindications using LLMs. We then used Llama 3.1 405B with a custom prompt to generate a list of thrombolysis contraindications from each note. Performance was evaluated using sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, and F1 score. Results: A total of 150 synthetic notes were generated using five different models: ChatGPT-4o, Llama 3.1 405B, Llama 3.1 70B, ChatGPT-4o mini, and Gemini 1.5 Flash. On average, each note contained 241.6 words (SD 110.7; range 80–549) and included 1.5 contraindications (SD 1.1; range 0–5). Our tool achieved a sensitivity of 90.9% (95% CI: 86.3%–94.3%), specificity of 99.2% (95% CI: 98.8%–99.5%), PPV of 87.7% (95% CI: 82.7%–91.7%), NPV of 99.4% (95% CI: 99.1%–99.6%), accuracy of 98.7% (95% CI: 98.2%–99.0%), and an F1 score of 0.892. Among the false positives, 24 (86%) were due to the inclusion of irrelevant contraindications, and 4 (14%) resulted from repetitive information. No hallucinations were observed. Conclusion: Our LLM-based tool may identify stroke thrombolysis contraindications from synthetic clinical notes with high sensitivity and PPV. Future studies will validate its performance using real EMR data and integrate it into acute stroke workflows to facilitate faster and safer thrombolysis decision-making. |
| format | Article |
| id | doaj-art-a261535fef634dac8240d91aed8212a5 |
| institution | Kabale University |
| issn | 1664-5456 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | Karger Publishers |
| record_format | Article |
| series | Cerebrovascular Diseases Extra |
| spelling | doaj-art-a261535fef634dac8240d91aed8212a52025-08-20T03:49:42ZengKarger PublishersCerebrovascular Diseases Extra1664-54562025-03-0115113013610.1159/000545317Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept StudyBing Yu ChenFares AntakiMarco GonzalezKen Uchinohttps://orcid.org/0000-0001-9468-4172Samer AlbahraScott RobertsonSidonie IbrikjiEric AubeAndrew RussmanMuhammad Shazam Hussain Introduction: Timely thrombolytic therapy improves outcomes in acute ischemic stroke. Manual chart review to screen for thrombolysis contraindications may be time-consuming and prone to errors. We developed and tested a large language model (LLM)-based tool to identify thrombolysis contraindications from clinical notes using synthetic data in a proof-of-concept study. Methods: We generated 150 synthetic clinical notes containing randomly assigned thrombolysis contraindications using LLMs. We then used Llama 3.1 405B with a custom prompt to generate a list of thrombolysis contraindications from each note. Performance was evaluated using sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, and F1 score. Results: A total of 150 synthetic notes were generated using five different models: ChatGPT-4o, Llama 3.1 405B, Llama 3.1 70B, ChatGPT-4o mini, and Gemini 1.5 Flash. On average, each note contained 241.6 words (SD 110.7; range 80–549) and included 1.5 contraindications (SD 1.1; range 0–5). Our tool achieved a sensitivity of 90.9% (95% CI: 86.3%–94.3%), specificity of 99.2% (95% CI: 98.8%–99.5%), PPV of 87.7% (95% CI: 82.7%–91.7%), NPV of 99.4% (95% CI: 99.1%–99.6%), accuracy of 98.7% (95% CI: 98.2%–99.0%), and an F1 score of 0.892. Among the false positives, 24 (86%) were due to the inclusion of irrelevant contraindications, and 4 (14%) resulted from repetitive information. No hallucinations were observed. Conclusion: Our LLM-based tool may identify stroke thrombolysis contraindications from synthetic clinical notes with high sensitivity and PPV. Future studies will validate its performance using real EMR data and integrate it into acute stroke workflows to facilitate faster and safer thrombolysis decision-making. https://karger.com/article/doi/10.1159/000545317 |
| spellingShingle | Bing Yu Chen Fares Antaki Marco Gonzalez Ken Uchino Samer Albahra Scott Robertson Sidonie Ibrikji Eric Aube Andrew Russman Muhammad Shazam Hussain Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept Study Cerebrovascular Diseases Extra |
| title | Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept Study |
| title_full | Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept Study |
| title_fullStr | Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept Study |
| title_full_unstemmed | Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept Study |
| title_short | Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept Study |
| title_sort | automated identification of stroke thrombolysis contraindications from synthetic clinical notes a proof of concept study |
| url | https://karger.com/article/doi/10.1159/000545317 |
| work_keys_str_mv | AT bingyuchen automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT faresantaki automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT marcogonzalez automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT kenuchino automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT sameralbahra automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT scottrobertson automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT sidonieibrikji automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT ericaube automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT andrewrussman automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy AT muhammadshazamhussain automatedidentificationofstrokethrombolysiscontraindicationsfromsyntheticclinicalnotesaproofofconceptstudy |