From Jargon to Clarity. Improving the Readability of Foot and Ankle Radiology Reports with An Artificial Intelligence Large Language Model
Category: Other; Ankle Introduction/Purpose: The purpose of this study was to evaluate the efficacy of an Artificial Intelligence Large Language Model (AI-LLM) at improving the readability foot and ankle orthopedic radiology reports. Methods: The radiology reports from 100 foot or ankle X-Rays, 100...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SAGE Publishing
2024-12-01
|
| Series: | Foot & Ankle Orthopaedics |
| Online Access: | https://doi.org/10.1177/2473011424S00301 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850242796519161856 |
|---|---|
| author | James J. Butler MB BCh BAO Michael Harrington MD Yixuan Tong MD Andrew Rosenbaum MD Alan P. Samsonov BS Raymond J. Walls MD, FRCS (Orth), MFSEM, FAAOS John G. Kennedy MD, MCh, MMSc, FFSEM, FRCS (Orth) |
| author_facet | James J. Butler MB BCh BAO Michael Harrington MD Yixuan Tong MD Andrew Rosenbaum MD Alan P. Samsonov BS Raymond J. Walls MD, FRCS (Orth), MFSEM, FAAOS John G. Kennedy MD, MCh, MMSc, FFSEM, FRCS (Orth) |
| author_sort | James J. Butler MB BCh BAO |
| collection | DOAJ |
| description | Category: Other; Ankle Introduction/Purpose: The purpose of this study was to evaluate the efficacy of an Artificial Intelligence Large Language Model (AI-LLM) at improving the readability foot and ankle orthopedic radiology reports. Methods: The radiology reports from 100 foot or ankle X-Rays, 100 computed tomography (CT) scans and 100 magnetic resonance imaging (MRI) scans were randomly sampled from the institution’s database. The following prompt command was inserted into the AI-LLM. “Explain this radiology report to a patient in layman's terms in the second person. [Report Text]”. The mean report length, Flesch reading ease score (FRES) and Flesch-Kincaid reading level (FKRL) were evaluated for both the original radiology report and the AI-LLM generated report. The accuracy of the information contained within the AI-LLM report was assessed via a 5-point Likert scale. Additionally, any “hallucinations” generated by the AI-LLM report were recorded. Results: There was a statistically significant improvement in mean FRES scores in the AI-LLM generated X-Ray report (33.8±6.8 to 72.7±5.4), CT report (27.8±4.6 to 67.5±4.9) and MRI report (20.3±7.2 to 66.9±3.9), p< 0.001. There was also a statistically significant improvement in mean FKRL scores in the AI-LLM generated X-Ray report (12.2±1.1 to 8.5±0.4), CT report (15.4±2.0 to 8.4±0.6) and MRI report (14.1±1.6 to 8.5±0.5), p< 0.001. Superior FRES scores were observed in the AI-LLM generated X-Ray report compared to the AI-LLM generated CT report and MRI report, p< 0.001. The mean Likert score for the AI-LLM generated X-Ray, CT and MRI report was 4.0±0.3, 3.9±0.4, and 3.9±0.4, respectively. The hallucination rate in the AI-LLM generated X-Ray report, CT report and MRI report was 4%, 7% and 6%, respectively. Conclusion: AI-LLM was an efficacious tool for improving the readability of foot and ankle radiological reports across multiple imaging modalities. Superior FRES scores together with superior Likert scores were observed in the X-Ray AI-LLM reports compared to the CT and MRI AI-LLM reports. This study demonstrates the potential use of AI-LLMs as a new patient-centric approach for enhancing patient understanding of their foot and ankle radiology reports. |
| format | Article |
| id | doaj-art-349c54c632d04bb09d64a95c4975f136 |
| institution | OA Journals |
| issn | 2473-0114 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | SAGE Publishing |
| record_format | Article |
| series | Foot & Ankle Orthopaedics |
| spelling | doaj-art-349c54c632d04bb09d64a95c4975f1362025-08-20T02:00:12ZengSAGE PublishingFoot & Ankle Orthopaedics2473-01142024-12-01910.1177/2473011424S00301From Jargon to Clarity. Improving the Readability of Foot and Ankle Radiology Reports with An Artificial Intelligence Large Language ModelJames J. Butler MB BCh BAOMichael Harrington MDYixuan Tong MDAndrew Rosenbaum MDAlan P. Samsonov BSRaymond J. Walls MD, FRCS (Orth), MFSEM, FAAOSJohn G. Kennedy MD, MCh, MMSc, FFSEM, FRCS (Orth)Category: Other; Ankle Introduction/Purpose: The purpose of this study was to evaluate the efficacy of an Artificial Intelligence Large Language Model (AI-LLM) at improving the readability foot and ankle orthopedic radiology reports. Methods: The radiology reports from 100 foot or ankle X-Rays, 100 computed tomography (CT) scans and 100 magnetic resonance imaging (MRI) scans were randomly sampled from the institution’s database. The following prompt command was inserted into the AI-LLM. “Explain this radiology report to a patient in layman's terms in the second person. [Report Text]”. The mean report length, Flesch reading ease score (FRES) and Flesch-Kincaid reading level (FKRL) were evaluated for both the original radiology report and the AI-LLM generated report. The accuracy of the information contained within the AI-LLM report was assessed via a 5-point Likert scale. Additionally, any “hallucinations” generated by the AI-LLM report were recorded. Results: There was a statistically significant improvement in mean FRES scores in the AI-LLM generated X-Ray report (33.8±6.8 to 72.7±5.4), CT report (27.8±4.6 to 67.5±4.9) and MRI report (20.3±7.2 to 66.9±3.9), p< 0.001. There was also a statistically significant improvement in mean FKRL scores in the AI-LLM generated X-Ray report (12.2±1.1 to 8.5±0.4), CT report (15.4±2.0 to 8.4±0.6) and MRI report (14.1±1.6 to 8.5±0.5), p< 0.001. Superior FRES scores were observed in the AI-LLM generated X-Ray report compared to the AI-LLM generated CT report and MRI report, p< 0.001. The mean Likert score for the AI-LLM generated X-Ray, CT and MRI report was 4.0±0.3, 3.9±0.4, and 3.9±0.4, respectively. The hallucination rate in the AI-LLM generated X-Ray report, CT report and MRI report was 4%, 7% and 6%, respectively. Conclusion: AI-LLM was an efficacious tool for improving the readability of foot and ankle radiological reports across multiple imaging modalities. Superior FRES scores together with superior Likert scores were observed in the X-Ray AI-LLM reports compared to the CT and MRI AI-LLM reports. This study demonstrates the potential use of AI-LLMs as a new patient-centric approach for enhancing patient understanding of their foot and ankle radiology reports.https://doi.org/10.1177/2473011424S00301 |
| spellingShingle | James J. Butler MB BCh BAO Michael Harrington MD Yixuan Tong MD Andrew Rosenbaum MD Alan P. Samsonov BS Raymond J. Walls MD, FRCS (Orth), MFSEM, FAAOS John G. Kennedy MD, MCh, MMSc, FFSEM, FRCS (Orth) From Jargon to Clarity. Improving the Readability of Foot and Ankle Radiology Reports with An Artificial Intelligence Large Language Model Foot & Ankle Orthopaedics |
| title | From Jargon to Clarity. Improving the Readability of Foot and Ankle Radiology Reports with An Artificial Intelligence Large Language Model |
| title_full | From Jargon to Clarity. Improving the Readability of Foot and Ankle Radiology Reports with An Artificial Intelligence Large Language Model |
| title_fullStr | From Jargon to Clarity. Improving the Readability of Foot and Ankle Radiology Reports with An Artificial Intelligence Large Language Model |
| title_full_unstemmed | From Jargon to Clarity. Improving the Readability of Foot and Ankle Radiology Reports with An Artificial Intelligence Large Language Model |
| title_short | From Jargon to Clarity. Improving the Readability of Foot and Ankle Radiology Reports with An Artificial Intelligence Large Language Model |
| title_sort | from jargon to clarity improving the readability of foot and ankle radiology reports with an artificial intelligence large language model |
| url | https://doi.org/10.1177/2473011424S00301 |
| work_keys_str_mv | AT jamesjbutlermbbchbao fromjargontoclarityimprovingthereadabilityoffootandankleradiologyreportswithanartificialintelligencelargelanguagemodel AT michaelharringtonmd fromjargontoclarityimprovingthereadabilityoffootandankleradiologyreportswithanartificialintelligencelargelanguagemodel AT yixuantongmd fromjargontoclarityimprovingthereadabilityoffootandankleradiologyreportswithanartificialintelligencelargelanguagemodel AT andrewrosenbaummd fromjargontoclarityimprovingthereadabilityoffootandankleradiologyreportswithanartificialintelligencelargelanguagemodel AT alanpsamsonovbs fromjargontoclarityimprovingthereadabilityoffootandankleradiologyreportswithanartificialintelligencelargelanguagemodel AT raymondjwallsmdfrcsorthmfsemfaaos fromjargontoclarityimprovingthereadabilityoffootandankleradiologyreportswithanartificialintelligencelargelanguagemodel AT johngkennedymdmchmmscffsemfrcsorth fromjargontoclarityimprovingthereadabilityoffootandankleradiologyreportswithanartificialintelligencelargelanguagemodel |