Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study
Objectives Identifying whether there is a traumatic intracranial bleed (ICB+) on head CT is critical for clinical care and research. Free text CT reports are unstructured and therefore must undergo time-consuming manual review. Existing artificial intelligence classification schemes are not optimise...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMJ Publishing Group
2025-07-01
|
| Series: | BMJ Health & Care Informatics |
| Online Access: | https://informatics.bmj.com/content/32/1/e101433.full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850077557067612160 |
|---|---|
| author | Stephen H Thomas Jason Pott Imogen Skene Ben Bloom Michael Cheetham Adrian Haimovich Raine Astin-Chamberlain Sophie L Williams Sandra Langsted |
| author_facet | Stephen H Thomas Jason Pott Imogen Skene Ben Bloom Michael Cheetham Adrian Haimovich Raine Astin-Chamberlain Sophie L Williams Sandra Langsted |
| author_sort | Stephen H Thomas |
| collection | DOAJ |
| description | Objectives Identifying whether there is a traumatic intracranial bleed (ICB+) on head CT is critical for clinical care and research. Free text CT reports are unstructured and therefore must undergo time-consuming manual review. Existing artificial intelligence classification schemes are not optimised for the emergency department endpoint of classification of ICB+ or ICB−. We sought to assess three methods for classifying CT reports: a text classification (TC) programme, a commercial natural language processing programme (Clinithink) and a generative pretrained transformer large language model (Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting (DECIPHER)-LLM).Methods Primary objective: determine the diagnostic classification performance of the dichotomous categorisation of each of the three approaches.Secondary objective: determine whether the LLM could achieve a substantial reduction in CT report review workload while maintaining 100% sensitivity.Anonymised radiology reports of head CT scans performed for trauma were manually labelled as ICB+/−. Training and validation sets were randomly created to train the TC and natural language processing models. Prompts were written to train the LLM.Results 898 reports were manually labelled. Sensitivity and specificity (95% CI)) of TC, Clinithink and DECIPHER-LLM (with probability of ICB set at 10%) were respectively 87.9% (76.7% to 95.0%) and 98.2% (96.3% to 99.3%), 75.9% (62.8% to 86.1%) and 96.2% (93.8% to 97.8%) and 100% (93.8% to 100%) and 97.4% (95.3% to 98.8%).With DECIPHER-LLM probability of ICB+ threshold of 10% set to identify CT reports requiring manual evaluation, CT reports requiring manual classification reduced by an estimated 385/449 cases (85.7% (95% CI 82.1% to 88.9%)) while maintaining 100% sensitivity.Discussion and conclusion DECIPHER-LLM outperformed other tested free-text classification methods. |
| format | Article |
| id | doaj-art-d9be3cbc7f7c4e4698db14334d707970 |
| institution | DOAJ |
| issn | 2632-1009 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | BMJ Publishing Group |
| record_format | Article |
| series | BMJ Health & Care Informatics |
| spelling | doaj-art-d9be3cbc7f7c4e4698db14334d7079702025-08-20T02:45:46ZengBMJ Publishing GroupBMJ Health & Care Informatics2632-10092025-07-0132110.1136/bmjhci-2025-101433Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER studyStephen H Thomas0Jason Pott1Imogen Skene2Ben Bloom3Michael Cheetham4Adrian Haimovich5Raine Astin-Chamberlain6Sophie L Williams7Sandra Langsted84 Queen Mary University of London Barts and The London School of Medicine and Dentistry, London, UK1 Emergency Department, Barts Health NHS Trust, London, UKQueen Mary University of London, London, UKEmergency Department, Royal London Hospital, Barts Health NHS Trust, London, UKEmergency Department, Barts Health NHS Trust, London, UKDepartment of Emergency Medicine, Harvard Medical School, Boston, Massachusetts, USABarts Health NHS Trust, London, UKBarts Life Sciences, Barts Health NHS Trust, London, UKDepartment of Emergency Medicine, Randers Regional Hospital, Randers, DenmarkObjectives Identifying whether there is a traumatic intracranial bleed (ICB+) on head CT is critical for clinical care and research. Free text CT reports are unstructured and therefore must undergo time-consuming manual review. Existing artificial intelligence classification schemes are not optimised for the emergency department endpoint of classification of ICB+ or ICB−. We sought to assess three methods for classifying CT reports: a text classification (TC) programme, a commercial natural language processing programme (Clinithink) and a generative pretrained transformer large language model (Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting (DECIPHER)-LLM).Methods Primary objective: determine the diagnostic classification performance of the dichotomous categorisation of each of the three approaches.Secondary objective: determine whether the LLM could achieve a substantial reduction in CT report review workload while maintaining 100% sensitivity.Anonymised radiology reports of head CT scans performed for trauma were manually labelled as ICB+/−. Training and validation sets were randomly created to train the TC and natural language processing models. Prompts were written to train the LLM.Results 898 reports were manually labelled. Sensitivity and specificity (95% CI)) of TC, Clinithink and DECIPHER-LLM (with probability of ICB set at 10%) were respectively 87.9% (76.7% to 95.0%) and 98.2% (96.3% to 99.3%), 75.9% (62.8% to 86.1%) and 96.2% (93.8% to 97.8%) and 100% (93.8% to 100%) and 97.4% (95.3% to 98.8%).With DECIPHER-LLM probability of ICB+ threshold of 10% set to identify CT reports requiring manual evaluation, CT reports requiring manual classification reduced by an estimated 385/449 cases (85.7% (95% CI 82.1% to 88.9%)) while maintaining 100% sensitivity.Discussion and conclusion DECIPHER-LLM outperformed other tested free-text classification methods.https://informatics.bmj.com/content/32/1/e101433.full |
| spellingShingle | Stephen H Thomas Jason Pott Imogen Skene Ben Bloom Michael Cheetham Adrian Haimovich Raine Astin-Chamberlain Sophie L Williams Sandra Langsted Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study BMJ Health & Care Informatics |
| title | Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study |
| title_full | Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study |
| title_fullStr | Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study |
| title_full_unstemmed | Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study |
| title_short | Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study |
| title_sort | digitalizing english language ct interpretation for positive haemorrhage evaluation reporting the decipher study |
| url | https://informatics.bmj.com/content/32/1/e101433.full |
| work_keys_str_mv | AT stephenhthomas digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy AT jasonpott digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy AT imogenskene digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy AT benbloom digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy AT michaelcheetham digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy AT adrianhaimovich digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy AT raineastinchamberlain digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy AT sophielwilliams digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy AT sandralangsted digitalizingenglishlanguagectinterpretationforpositivehaemorrhageevaluationreportingthedecipherstudy |