A public benchmark for human performance in the detection of focal cortical dysplasia
Abstract Objective This study aims to report human performance in the detection of Focal Cortical Dysplasias (FCDs) using an openly available dataset. Additionally, it defines a subset of this data as a “difficult” test set to establish a public baseline benchmark against which new methods for autom...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2025-06-01
|
| Series: | Epilepsia Open |
| Subjects: | |
| Online Access: | https://doi.org/10.1002/epi4.70028 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849330293510504448 |
|---|---|
| author | Lennart Walger Matthias H. Schmitz Tobias Bauer David Kügler Fabiane Schuch Christophe Arendt Tobias Baumgartner Johannes Birkenheier Valeri Borger Christoph Endler Franziska Grau Christian Immanuel Markus Kölle Patrick Kupczyk Asadeh Lakghomi Sarah Mackert Elisabeth Neuhaus Julia Nordsiek Anna‐Maria Odenthal Karmele Olaciregui Dague Laura Ostermann Jan Pukropski Attila Racz Klaus von derRopp Frederic Carsten Schmeel Felix Schrader Aileen Sitter Alexander Unruh‐Pinheiro Marilia Voigt Martin Vychopen Philip vonWedel Randi vonWrede Ulrike Attenberger Hartmut Vatter Alexandra Philipsen Albert Becker Martin Reuter Elke Hattingen Alexander Radbruch Rainer Surges Theodor Rüber |
| author_facet | Lennart Walger Matthias H. Schmitz Tobias Bauer David Kügler Fabiane Schuch Christophe Arendt Tobias Baumgartner Johannes Birkenheier Valeri Borger Christoph Endler Franziska Grau Christian Immanuel Markus Kölle Patrick Kupczyk Asadeh Lakghomi Sarah Mackert Elisabeth Neuhaus Julia Nordsiek Anna‐Maria Odenthal Karmele Olaciregui Dague Laura Ostermann Jan Pukropski Attila Racz Klaus von derRopp Frederic Carsten Schmeel Felix Schrader Aileen Sitter Alexander Unruh‐Pinheiro Marilia Voigt Martin Vychopen Philip vonWedel Randi vonWrede Ulrike Attenberger Hartmut Vatter Alexandra Philipsen Albert Becker Martin Reuter Elke Hattingen Alexander Radbruch Rainer Surges Theodor Rüber |
| author_sort | Lennart Walger |
| collection | DOAJ |
| description | Abstract Objective This study aims to report human performance in the detection of Focal Cortical Dysplasias (FCDs) using an openly available dataset. Additionally, it defines a subset of this data as a “difficult” test set to establish a public baseline benchmark against which new methods for automated FCD detection can be evaluated. Methods The performance of 28 human readers with varying levels of expertise in detecting FCDs was originally analyzed using 146 subjects (not all of which are openly available), we analyzed the openly available subset of 85 cases. Performance was measured based on the overlap between predicted regions of interest (ROIs) and ground‐truth lesion masks, using the Dice‐Soerensen coefficient (DSC). The benchmark test set was chosen to consist of 15 subjects most predictive for human performance and 13 subjects identified by at most 3 of the 28 readers. Results Expert readers achieved an average detection rate of 68%, compared to 45% for non‐experts and 27% for laypersons. Neuroradiologists detected the highest percentage of lesions (64%), while psychiatrists detected the least (34%). Neurosurgeons had the highest ROI sensitivity (0.70), and psychiatrists had the highest ROI precision (0.78). The benchmark test set revealed an expert detection rate of 49%. Significance Reporting human performance in FCD detection provides a critical baseline for assessing the effectiveness of automated detection methods in a clinically relevant context. The defined benchmark test set serves as a useful indicator for evaluating advancements in computer‐aided FCD detection approaches. Plain Language Summary Focal cortical dysplasias (FCDs) are malformations of cortical development and one of the most common causes of drug‐resistant focal epilepsy. Once found, FCDs can be neurosurgically resected, which leads to seizure freedom in many cases. However, FCDs are difficult to detect in the visual assessment of magnetic resonance imaging. A myriad of algorithms for automated FCD detection have been developed, but their true clinical value remains unclear since there is no benchmark dataset for evaluation and comparison to human performance. Here, we use human FCD detection performance to define a benchmark dataset with which new methods for automated detection can be evaluated. |
| format | Article |
| id | doaj-art-5e20f184f5d14b218036bb109ddaf5e4 |
| institution | Kabale University |
| issn | 2470-9239 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | Wiley |
| record_format | Article |
| series | Epilepsia Open |
| spelling | doaj-art-5e20f184f5d14b218036bb109ddaf5e42025-08-20T03:46:58ZengWileyEpilepsia Open2470-92392025-06-0110377878610.1002/epi4.70028A public benchmark for human performance in the detection of focal cortical dysplasiaLennart Walger0Matthias H. Schmitz1Tobias Bauer2David Kügler3Fabiane Schuch4Christophe Arendt5Tobias Baumgartner6Johannes Birkenheier7Valeri Borger8Christoph Endler9Franziska Grau10Christian Immanuel11Markus Kölle12Patrick Kupczyk13Asadeh Lakghomi14Sarah Mackert15Elisabeth Neuhaus16Julia Nordsiek17Anna‐Maria Odenthal18Karmele Olaciregui Dague19Laura Ostermann20Jan Pukropski21Attila Racz22Klaus von derRopp23Frederic Carsten Schmeel24Felix Schrader25Aileen Sitter26Alexander Unruh‐Pinheiro27Marilia Voigt28Martin Vychopen29Philip vonWedel30Randi vonWrede31Ulrike Attenberger32Hartmut Vatter33Alexandra Philipsen34Albert Becker35Martin Reuter36Elke Hattingen37Alexander Radbruch38Rainer Surges39Theodor Rüber40Department of Neuroradiology University Hospital Bonn Bonn GermanyDepartment of Neuroradiology University Hospital Bonn Bonn GermanyDepartment of Neuroradiology University Hospital Bonn Bonn GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Neuroradiology Goethe University Frankfurt Frankfurt GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Neurology University Hospital Bonn Bonn GermanyDepartment of Neurosurgery University Hospital Bonn Bonn GermanyDepartment of Diagnostic and Interventional Radiology University Hospital Bonn Bonn GermanyDepartment of Neuroradiology University Hospital Bonn Bonn GermanyDepartment of Diagnostic and Interventional Radiology University Hospital Bonn Bonn GermanyDepartment of Psychiatry and Psychotherapy University Hospital Bonn Bonn GermanyDepartment of Diagnostic and Interventional Radiology University Hospital Bonn Bonn GermanyDepartment of Neuroradiology University Hospital Bonn Bonn GermanyDepartment of Psychiatry and Psychotherapy University Hospital Bonn Bonn GermanyDepartment of Neuroradiology Goethe University Frankfurt Frankfurt GermanyDepartment of Neurology University Hospital Bonn Bonn GermanyDepartment of Diagnostic and Interventional Radiology University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Neuroradiology University Hospital Bonn Bonn GermanyDepartment of Psychiatry and Psychotherapy University Hospital Bonn Bonn GermanyDepartment of Psychiatry and Psychotherapy University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Diagnostic and Interventional Radiology University Hospital Bonn Bonn GermanyDepartment of Neurosurgery University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Diagnostic and Interventional Radiology University Hospital Bonn Bonn GermanyDepartment of Neurosurgery University Hospital Bonn Bonn GermanyDepartment of Psychiatry and Psychotherapy University Hospital Bonn Bonn GermanyDepartment of Neuropathology University Hospital Bonn Bonn GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyDepartment of Neuroradiology Goethe University Frankfurt Frankfurt GermanyDepartment of Neuroradiology University Hospital Bonn Bonn GermanyDepartment of Epileptology University Hospital Bonn Bonn GermanyDepartment of Neuroradiology University Hospital Bonn Bonn GermanyAbstract Objective This study aims to report human performance in the detection of Focal Cortical Dysplasias (FCDs) using an openly available dataset. Additionally, it defines a subset of this data as a “difficult” test set to establish a public baseline benchmark against which new methods for automated FCD detection can be evaluated. Methods The performance of 28 human readers with varying levels of expertise in detecting FCDs was originally analyzed using 146 subjects (not all of which are openly available), we analyzed the openly available subset of 85 cases. Performance was measured based on the overlap between predicted regions of interest (ROIs) and ground‐truth lesion masks, using the Dice‐Soerensen coefficient (DSC). The benchmark test set was chosen to consist of 15 subjects most predictive for human performance and 13 subjects identified by at most 3 of the 28 readers. Results Expert readers achieved an average detection rate of 68%, compared to 45% for non‐experts and 27% for laypersons. Neuroradiologists detected the highest percentage of lesions (64%), while psychiatrists detected the least (34%). Neurosurgeons had the highest ROI sensitivity (0.70), and psychiatrists had the highest ROI precision (0.78). The benchmark test set revealed an expert detection rate of 49%. Significance Reporting human performance in FCD detection provides a critical baseline for assessing the effectiveness of automated detection methods in a clinically relevant context. The defined benchmark test set serves as a useful indicator for evaluating advancements in computer‐aided FCD detection approaches. Plain Language Summary Focal cortical dysplasias (FCDs) are malformations of cortical development and one of the most common causes of drug‐resistant focal epilepsy. Once found, FCDs can be neurosurgically resected, which leads to seizure freedom in many cases. However, FCDs are difficult to detect in the visual assessment of magnetic resonance imaging. A myriad of algorithms for automated FCD detection have been developed, but their true clinical value remains unclear since there is no benchmark dataset for evaluation and comparison to human performance. Here, we use human FCD detection performance to define a benchmark dataset with which new methods for automated detection can be evaluated.https://doi.org/10.1002/epi4.70028artificial intelligencecomputer‐aided detectionhuman performancereader study |
| spellingShingle | Lennart Walger Matthias H. Schmitz Tobias Bauer David Kügler Fabiane Schuch Christophe Arendt Tobias Baumgartner Johannes Birkenheier Valeri Borger Christoph Endler Franziska Grau Christian Immanuel Markus Kölle Patrick Kupczyk Asadeh Lakghomi Sarah Mackert Elisabeth Neuhaus Julia Nordsiek Anna‐Maria Odenthal Karmele Olaciregui Dague Laura Ostermann Jan Pukropski Attila Racz Klaus von derRopp Frederic Carsten Schmeel Felix Schrader Aileen Sitter Alexander Unruh‐Pinheiro Marilia Voigt Martin Vychopen Philip vonWedel Randi vonWrede Ulrike Attenberger Hartmut Vatter Alexandra Philipsen Albert Becker Martin Reuter Elke Hattingen Alexander Radbruch Rainer Surges Theodor Rüber A public benchmark for human performance in the detection of focal cortical dysplasia Epilepsia Open artificial intelligence computer‐aided detection human performance reader study |
| title | A public benchmark for human performance in the detection of focal cortical dysplasia |
| title_full | A public benchmark for human performance in the detection of focal cortical dysplasia |
| title_fullStr | A public benchmark for human performance in the detection of focal cortical dysplasia |
| title_full_unstemmed | A public benchmark for human performance in the detection of focal cortical dysplasia |
| title_short | A public benchmark for human performance in the detection of focal cortical dysplasia |
| title_sort | public benchmark for human performance in the detection of focal cortical dysplasia |
| topic | artificial intelligence computer‐aided detection human performance reader study |
| url | https://doi.org/10.1002/epi4.70028 |
| work_keys_str_mv | AT lennartwalger apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT matthiashschmitz apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT tobiasbauer apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT davidkugler apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT fabianeschuch apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT christophearendt apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT tobiasbaumgartner apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT johannesbirkenheier apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT valeriborger apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT christophendler apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT franziskagrau apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT christianimmanuel apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT markuskolle apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT patrickkupczyk apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT asadehlakghomi apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT sarahmackert apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT elisabethneuhaus apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT julianordsiek apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT annamariaodenthal apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT karmeleolacireguidague apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT lauraostermann apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT janpukropski apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT attilaracz apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT klausvonderropp apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT fredericcarstenschmeel apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT felixschrader apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT aileensitter apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT alexanderunruhpinheiro apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT mariliavoigt apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT martinvychopen apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT philipvonwedel apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT randivonwrede apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT ulrikeattenberger apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT hartmutvatter apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT alexandraphilipsen apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT albertbecker apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT martinreuter apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT elkehattingen apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT alexanderradbruch apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT rainersurges apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT theodorruber apublicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT lennartwalger publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT matthiashschmitz publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT tobiasbauer publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT davidkugler publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT fabianeschuch publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT christophearendt publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT tobiasbaumgartner publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT johannesbirkenheier publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT valeriborger publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT christophendler publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT franziskagrau publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT christianimmanuel publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT markuskolle publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT patrickkupczyk publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT asadehlakghomi publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT sarahmackert publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT elisabethneuhaus publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT julianordsiek publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT annamariaodenthal publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT karmeleolacireguidague publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT lauraostermann publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT janpukropski publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT attilaracz publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT klausvonderropp publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT fredericcarstenschmeel publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT felixschrader publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT aileensitter publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT alexanderunruhpinheiro publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT mariliavoigt publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT martinvychopen publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT philipvonwedel publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT randivonwrede publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT ulrikeattenberger publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT hartmutvatter publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT alexandraphilipsen publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT albertbecker publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT martinreuter publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT elkehattingen publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT alexanderradbruch publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT rainersurges publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia AT theodorruber publicbenchmarkforhumanperformanceinthedetectionoffocalcorticaldysplasia |