Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting.
<h4>Introduction</h4>We examined the design, analysis and reporting in multi-reader multi-case (MRMC) research studies using the area under the receiver-operating curve (ROC AUC) as a measure of diagnostic performance.<h4>Methods</h4>We performed a systematic literature revie...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2014-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0116018 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849412043434622976 |
|---|---|
| author | Thaworn Dendumrongsup Andrew A Plumb Steve Halligan Thomas R Fanshawe Douglas G Altman Susan Mallett |
| author_facet | Thaworn Dendumrongsup Andrew A Plumb Steve Halligan Thomas R Fanshawe Douglas G Altman Susan Mallett |
| author_sort | Thaworn Dendumrongsup |
| collection | DOAJ |
| description | <h4>Introduction</h4>We examined the design, analysis and reporting in multi-reader multi-case (MRMC) research studies using the area under the receiver-operating curve (ROC AUC) as a measure of diagnostic performance.<h4>Methods</h4>We performed a systematic literature review from 2005 to 2013 inclusive to identify a minimum 50 studies. Articles of diagnostic test accuracy in humans were identified via their citation of key methodological articles dealing with MRMC ROC AUC. Two researchers in consensus then extracted information from primary articles relating to study characteristics and design, methods for reporting study outcomes, model fitting, model assumptions, presentation of results, and interpretation of findings. Results were summarized and presented with a descriptive analysis.<h4>Results</h4>Sixty-four full papers were retrieved from 475 identified citations and ultimately 49 articles describing 51 studies were reviewed and extracted. Radiological imaging was the index test in all. Most studies focused on lesion detection vs. characterization and used less than 10 readers. Only 6 (12%) studies trained readers in advance to use the confidence scale used to build the ROC curve. Overall, description of confidence scores, the ROC curve and its analysis was often incomplete. For example, 21 (41%) studies presented no ROC curve and only 3 (6%) described the distribution of confidence scores. Of 30 studies presenting curves, only 4 (13%) presented the data points underlying the curve, thereby allowing assessment of extrapolation. The mean change in AUC was 0.05 (-0.05 to 0.28). Non-significant change in AUC was attributed to underpowering rather than the diagnostic test failing to improve diagnostic accuracy.<h4>Conclusions</h4>Data reporting in MRMC studies using ROC AUC as an outcome measure is frequently incomplete, hampering understanding of methods and the reliability of results and study conclusions. Authors using this analysis should be encouraged to provide a full description of their methods and results. |
| format | Article |
| id | doaj-art-d86a6b606b81474cbb5630902e18afee |
| institution | Kabale University |
| issn | 1932-6203 |
| language | English |
| publishDate | 2014-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-d86a6b606b81474cbb5630902e18afee2025-08-20T03:34:33ZengPublic Library of Science (PLoS)PLoS ONE1932-62032014-01-01912e11601810.1371/journal.pone.0116018Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting.Thaworn DendumrongsupAndrew A PlumbSteve HalliganThomas R FanshaweDouglas G AltmanSusan Mallett<h4>Introduction</h4>We examined the design, analysis and reporting in multi-reader multi-case (MRMC) research studies using the area under the receiver-operating curve (ROC AUC) as a measure of diagnostic performance.<h4>Methods</h4>We performed a systematic literature review from 2005 to 2013 inclusive to identify a minimum 50 studies. Articles of diagnostic test accuracy in humans were identified via their citation of key methodological articles dealing with MRMC ROC AUC. Two researchers in consensus then extracted information from primary articles relating to study characteristics and design, methods for reporting study outcomes, model fitting, model assumptions, presentation of results, and interpretation of findings. Results were summarized and presented with a descriptive analysis.<h4>Results</h4>Sixty-four full papers were retrieved from 475 identified citations and ultimately 49 articles describing 51 studies were reviewed and extracted. Radiological imaging was the index test in all. Most studies focused on lesion detection vs. characterization and used less than 10 readers. Only 6 (12%) studies trained readers in advance to use the confidence scale used to build the ROC curve. Overall, description of confidence scores, the ROC curve and its analysis was often incomplete. For example, 21 (41%) studies presented no ROC curve and only 3 (6%) described the distribution of confidence scores. Of 30 studies presenting curves, only 4 (13%) presented the data points underlying the curve, thereby allowing assessment of extrapolation. The mean change in AUC was 0.05 (-0.05 to 0.28). Non-significant change in AUC was attributed to underpowering rather than the diagnostic test failing to improve diagnostic accuracy.<h4>Conclusions</h4>Data reporting in MRMC studies using ROC AUC as an outcome measure is frequently incomplete, hampering understanding of methods and the reliability of results and study conclusions. Authors using this analysis should be encouraged to provide a full description of their methods and results.https://doi.org/10.1371/journal.pone.0116018 |
| spellingShingle | Thaworn Dendumrongsup Andrew A Plumb Steve Halligan Thomas R Fanshawe Douglas G Altman Susan Mallett Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting. PLoS ONE |
| title | Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting. |
| title_full | Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting. |
| title_fullStr | Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting. |
| title_full_unstemmed | Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting. |
| title_short | Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting. |
| title_sort | multi reader multi case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy systematic review with a focus on quality of data reporting |
| url | https://doi.org/10.1371/journal.pone.0116018 |
| work_keys_str_mv | AT thaworndendumrongsup multireadermulticasestudiesusingtheareaunderthereceiveroperatorcharacteristiccurveasameasureofdiagnosticaccuracysystematicreviewwithafocusonqualityofdatareporting AT andrewaplumb multireadermulticasestudiesusingtheareaunderthereceiveroperatorcharacteristiccurveasameasureofdiagnosticaccuracysystematicreviewwithafocusonqualityofdatareporting AT stevehalligan multireadermulticasestudiesusingtheareaunderthereceiveroperatorcharacteristiccurveasameasureofdiagnosticaccuracysystematicreviewwithafocusonqualityofdatareporting AT thomasrfanshawe multireadermulticasestudiesusingtheareaunderthereceiveroperatorcharacteristiccurveasameasureofdiagnosticaccuracysystematicreviewwithafocusonqualityofdatareporting AT douglasgaltman multireadermulticasestudiesusingtheareaunderthereceiveroperatorcharacteristiccurveasameasureofdiagnosticaccuracysystematicreviewwithafocusonqualityofdatareporting AT susanmallett multireadermulticasestudiesusingtheareaunderthereceiveroperatorcharacteristiccurveasameasureofdiagnosticaccuracysystematicreviewwithafocusonqualityofdatareporting |