Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling Scheme
<b>Objective:</b> Reliable reading and annotation of chest X-ray (CXR) images are essential for both clinical decision-making and AI model development. While most of the literature emphasizes pulmonary findings, this study evaluates the consistency and reliability of annotations for extr...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-04-01
|
| Series: | Diagnostics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2075-4418/15/7/902 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849730663874297856 |
|---|---|
| author | Lea Marie Pehrson Dana Li Alyas Mayar Marco Fraccaro Rasmus Bonnevie Peter Jagd Sørensen Alexander Malcom Rykkje Tobias Thostrup Andersen Henrik Steglich-Arnholm Dorte Marianne Rohde Stærk Lotte Borgwardt Sune Darkner Jonathan Frederik Carlsen Michael Bachmann Nielsen Silvia Ingala |
| author_facet | Lea Marie Pehrson Dana Li Alyas Mayar Marco Fraccaro Rasmus Bonnevie Peter Jagd Sørensen Alexander Malcom Rykkje Tobias Thostrup Andersen Henrik Steglich-Arnholm Dorte Marianne Rohde Stærk Lotte Borgwardt Sune Darkner Jonathan Frederik Carlsen Michael Bachmann Nielsen Silvia Ingala |
| author_sort | Lea Marie Pehrson |
| collection | DOAJ |
| description | <b>Objective:</b> Reliable reading and annotation of chest X-ray (CXR) images are essential for both clinical decision-making and AI model development. While most of the literature emphasizes pulmonary findings, this study evaluates the consistency and reliability of annotations for extrapulmonary findings, using a labelling scheme. <b>Methods:</b> Six clinicians with varying experience levels (novice, intermediate, and experienced) annotated 100 CXR images using a diagnostic labelling scheme, in two rounds, separated by a three-week washout period. Annotation consistency was assessed using Randolph’s free-marginal kappa (RK), prevalence- and bias-adjusted kappa (PABAK), proportion positive agreement (PPA), and proportion negative agreement (PNA). Pairwise comparisons and the McNemar’s test were conducted to assess inter-reader and intra-reader agreement. <b>Results:</b> PABAK values indicated high overall grouped labelling agreement (novice: 0.86, intermediate: 0.90, experienced: 0.91). PNA values demonstrated strong agreement on negative findings, while PPA values showed moderate-to-low consistency in positive findings. Significant differences in specific agreement emerged between novice and experienced clinicians for eight labels, but there were no significant variations in RK across experience levels. The McNemar’s test confirmed annotation stability between rounds. <b>Conclusions:</b> This study demonstrates that clinician annotations of extrapulmonary findings in CXR are consistent and reliable across different experience levels using a pre-defined diagnostic labelling scheme. These insights aid in optimizing training strategies for both clinicians and AI models. |
| format | Article |
| id | doaj-art-a2f7fe3c9f07466e8e23c1b6d79ed3b3 |
| institution | DOAJ |
| issn | 2075-4418 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Diagnostics |
| spelling | doaj-art-a2f7fe3c9f07466e8e23c1b6d79ed3b32025-08-20T03:08:47ZengMDPI AGDiagnostics2075-44182025-04-0115790210.3390/diagnostics15070902Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling SchemeLea Marie Pehrson0Dana Li1Alyas Mayar2Marco Fraccaro3Rasmus Bonnevie4Peter Jagd Sørensen5Alexander Malcom Rykkje6Tobias Thostrup Andersen7Henrik Steglich-Arnholm8Dorte Marianne Rohde Stærk9Lotte Borgwardt10Sune Darkner11Jonathan Frederik Carlsen12Michael Bachmann Nielsen13Silvia Ingala14Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkUnumed Aps, 1055 Copenhagen, DenmarkUnumed Aps, 1055 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Computer Science, University of Copenhagen, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, DenmarkDepartment of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100 Copenhagen, Denmark<b>Objective:</b> Reliable reading and annotation of chest X-ray (CXR) images are essential for both clinical decision-making and AI model development. While most of the literature emphasizes pulmonary findings, this study evaluates the consistency and reliability of annotations for extrapulmonary findings, using a labelling scheme. <b>Methods:</b> Six clinicians with varying experience levels (novice, intermediate, and experienced) annotated 100 CXR images using a diagnostic labelling scheme, in two rounds, separated by a three-week washout period. Annotation consistency was assessed using Randolph’s free-marginal kappa (RK), prevalence- and bias-adjusted kappa (PABAK), proportion positive agreement (PPA), and proportion negative agreement (PNA). Pairwise comparisons and the McNemar’s test were conducted to assess inter-reader and intra-reader agreement. <b>Results:</b> PABAK values indicated high overall grouped labelling agreement (novice: 0.86, intermediate: 0.90, experienced: 0.91). PNA values demonstrated strong agreement on negative findings, while PPA values showed moderate-to-low consistency in positive findings. Significant differences in specific agreement emerged between novice and experienced clinicians for eight labels, but there were no significant variations in RK across experience levels. The McNemar’s test confirmed annotation stability between rounds. <b>Conclusions:</b> This study demonstrates that clinician annotations of extrapulmonary findings in CXR are consistent and reliable across different experience levels using a pre-defined diagnostic labelling scheme. These insights aid in optimizing training strategies for both clinicians and AI models.https://www.mdpi.com/2075-4418/15/7/902annotationextrapulmonary findingsconsistencyartificial intelligence |
| spellingShingle | Lea Marie Pehrson Dana Li Alyas Mayar Marco Fraccaro Rasmus Bonnevie Peter Jagd Sørensen Alexander Malcom Rykkje Tobias Thostrup Andersen Henrik Steglich-Arnholm Dorte Marianne Rohde Stærk Lotte Borgwardt Sune Darkner Jonathan Frederik Carlsen Michael Bachmann Nielsen Silvia Ingala Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling Scheme Diagnostics annotation extrapulmonary findings consistency artificial intelligence |
| title | Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling Scheme |
| title_full | Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling Scheme |
| title_fullStr | Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling Scheme |
| title_full_unstemmed | Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling Scheme |
| title_short | Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling Scheme |
| title_sort | clinicians agreement on extrapulmonary radiographic findings in chest x rays using a diagnostic labelling scheme |
| topic | annotation extrapulmonary findings consistency artificial intelligence |
| url | https://www.mdpi.com/2075-4418/15/7/902 |
| work_keys_str_mv | AT leamariepehrson cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT danali cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT alyasmayar cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT marcofraccaro cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT rasmusbonnevie cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT peterjagdsørensen cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT alexandermalcomrykkje cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT tobiasthostrupandersen cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT henriksteglicharnholm cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT dortemariannerohdestærk cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT lotteborgwardt cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT sunedarkner cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT jonathanfrederikcarlsen cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT michaelbachmannnielsen cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme AT silviaingala cliniciansagreementonextrapulmonaryradiographicfindingsinchestxraysusingadiagnosticlabellingscheme |