Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors
Abstract Accurate melanoma diagnosis is crucial for patient outcomes and reliability of AI diagnostic tools. We assess interrater variability among eight expert pathologists reviewing histopathological images and clinical metadata of 792 melanoma-suspicious lesions prospectively collected at eight G...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-01-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-025-56160-x |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832594581562064896 |
|---|---|
| author | Sarah Haggenmüller Christoph Wies Julia Abels Jana T. Winterstein Lukas Heinlein Carina Nogueira Garcia Jochen S. Utikal Sebastian A. Wohlfeil Friedegund Meier Sarah Hobelsberger Frank F. Gellrich Mildred Sergon Axel Hauschild Lars E. French Lucie Heinzerling Justin G. Schlager Kamran Ghoreschi Max Schlaak Franz J. Hilke Gabriela Poch Sören Korsing Cosimo Sarfert Carola Berking Markus V. Heppt Michael Erdmann Sebastian Haferkamp Konstantin Drexler Dirk Schadendorf Wiebke Sondermann Matthias Goebeler Bastian Schilling Jakob Nikolas Kather Stefan Fröhling Mar Llamas-Velasco Luis C. Requena Gerardo Ferrara Maite Fernandez-Figueras Sylvie Fraitag Cornelia S. L. Müller Hans Starz Heinz Kutzner Raymond Barnhill Richard Carr Kenneth S. Resnik Stephan Alexander Braun Tim Holland-Letz Titus J. Brinker |
| author_facet | Sarah Haggenmüller Christoph Wies Julia Abels Jana T. Winterstein Lukas Heinlein Carina Nogueira Garcia Jochen S. Utikal Sebastian A. Wohlfeil Friedegund Meier Sarah Hobelsberger Frank F. Gellrich Mildred Sergon Axel Hauschild Lars E. French Lucie Heinzerling Justin G. Schlager Kamran Ghoreschi Max Schlaak Franz J. Hilke Gabriela Poch Sören Korsing Cosimo Sarfert Carola Berking Markus V. Heppt Michael Erdmann Sebastian Haferkamp Konstantin Drexler Dirk Schadendorf Wiebke Sondermann Matthias Goebeler Bastian Schilling Jakob Nikolas Kather Stefan Fröhling Mar Llamas-Velasco Luis C. Requena Gerardo Ferrara Maite Fernandez-Figueras Sylvie Fraitag Cornelia S. L. Müller Hans Starz Heinz Kutzner Raymond Barnhill Richard Carr Kenneth S. Resnik Stephan Alexander Braun Tim Holland-Letz Titus J. Brinker |
| author_sort | Sarah Haggenmüller |
| collection | DOAJ |
| description | Abstract Accurate melanoma diagnosis is crucial for patient outcomes and reliability of AI diagnostic tools. We assess interrater variability among eight expert pathologists reviewing histopathological images and clinical metadata of 792 melanoma-suspicious lesions prospectively collected at eight German hospitals. Moreover, we provide access to the largest panel-validated dataset featuring dermoscopic and histopathological images with metadata. Complete agreement is achieved in 53.5% of cases (424/792), and a majority vote ( ≥ five pathologists) in 90.9% (720/792). Considerable discordance is observed for non-invasive melanomas (complete agreement in only 10/73 cases). The expert panel disagrees with the local pathologists’ and dermatologists’ diagnoses in 14.9% and 33.5% of cases, respectively. This variability highlights the diagnostic challenges of early-stage melanomas and the need to reconsider how ground truth is established in routine care and AI research. Including at least two pathologists or virtual panels may contribute to more consistent diagnostic results. |
| format | Article |
| id | doaj-art-db350769110441bebee694db13fbb5b0 |
| institution | Kabale University |
| issn | 2041-1723 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj-art-db350769110441bebee694db13fbb5b02025-01-19T12:30:30ZengNature PortfolioNature Communications2041-17232025-01-011611810.1038/s41467-025-56160-xDiscordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumorsSarah Haggenmüller0Christoph Wies1Julia Abels2Jana T. Winterstein3Lukas Heinlein4Carina Nogueira Garcia5Jochen S. Utikal6Sebastian A. Wohlfeil7Friedegund Meier8Sarah Hobelsberger9Frank F. Gellrich10Mildred Sergon11Axel Hauschild12Lars E. French13Lucie Heinzerling14Justin G. Schlager15Kamran Ghoreschi16Max Schlaak17Franz J. Hilke18Gabriela Poch19Sören Korsing20Cosimo Sarfert21Carola Berking22Markus V. Heppt23Michael Erdmann24Sebastian Haferkamp25Konstantin Drexler26Dirk Schadendorf27Wiebke Sondermann28Matthias Goebeler29Bastian Schilling30Jakob Nikolas Kather31Stefan Fröhling32Mar Llamas-Velasco33Luis C. Requena34Gerardo Ferrara35Maite Fernandez-Figueras36Sylvie Fraitag37Cornelia S. L. Müller38Hans Starz39Heinz Kutzner40Raymond Barnhill41Richard Carr42Kenneth S. Resnik43Stephan Alexander Braun44Tim Holland-Letz45Titus J. Brinker46Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ)Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ)Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ)Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ)Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ)Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ)Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of HeidelbergDepartment of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of HeidelbergSkin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität DresdenSkin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität DresdenSkin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität DresdenInstitute of Pathology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität DresdenDepartment of Dermatology, University Hospital (UKSH)Department of Dermatology and Allergy, University Hospital, LMU MunichDepartment of Dermatology and Allergy, University Hospital, LMU MunichDepartment of Dermatology and Allergy, University Hospital, LMU MunichDepartment of Dermatology, Venereology and Allergology, Charité – Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu BerlinDepartment of Dermatology, Venereology and Allergology, Charité – Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu BerlinDepartment of Dermatology, Venereology and Allergology, Charité – Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu BerlinDepartment of Dermatology, Venereology and Allergology, Charité – Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu BerlinDepartment of Dermatology, Venereology and Allergology, Charité – Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu BerlinDepartment of Dermatology, Venereology and Allergology, Charité – Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu BerlinDepartment of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen – European Metropolitan Region Nürnberg, CCC Alliance WERADepartment of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen – European Metropolitan Region Nürnberg, CCC Alliance WERADepartment of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen – European Metropolitan Region Nürnberg, CCC Alliance WERADepartment of Dermatology, University Hospital RegensburgDepartment of Dermatology, University Hospital RegensburgDepartment of Dermatology, Venereology and Allergology, University Hospital EssenDepartment of Dermatology, Venereology and Allergology, University Hospital EssenDepartment of Dermatology, Venereology and Allergology, University Hospital Würzburg and National Center for Tumor Diseases (NCT) WERA WürzburgGoethe University Frankfurt, University Hospital, Department of DermatologyElse Kroener Fresenius Center for Digital Health, Technical University DresdenDepartment of Translational Medical Oncology, National Center for Tumor Diseases (NCT) Heidelberg and German Cancer Research Center (DKFZ)Department of Dermatology, University Hospital La PrincesaDermatology Department, Fundación Jiménez Díaz, Autonomous University of Madrid, Madrid, Spain; Anatomic Pathology Service, Fundación Jiménez DíazAnatomic Pathology and Cytopathology Unit, Istituto Nazionale Tumori I.R.C.C.S. Fondazione ‘G. Pascale’University General Hospital of Catalonia, Grupo Quironsalud, International University of Catalonia, Sant Cugat del VallésPathology department, Necker-Enfants Malades Hospital, Université Paris-Cité, Assistance Publique des Hopitaux de ParisCenter for Histology, Cytology and Molecular DiagnosticsDermpath MünchenDermatopathology FriedrichshafenDepartments of Pathology and Translational Research, Institut CurieDepartment of Pathology, Warwick HospitalInstitute for DermatopathologyDepartment of Dermatology, University of Münster, Münster, Germany; Department of Dermatology, Medical Faculty, Heinrich-Heine-UniversityDepartment of Biostatistics, German Cancer Research Center (DKFZ)Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ)Abstract Accurate melanoma diagnosis is crucial for patient outcomes and reliability of AI diagnostic tools. We assess interrater variability among eight expert pathologists reviewing histopathological images and clinical metadata of 792 melanoma-suspicious lesions prospectively collected at eight German hospitals. Moreover, we provide access to the largest panel-validated dataset featuring dermoscopic and histopathological images with metadata. Complete agreement is achieved in 53.5% of cases (424/792), and a majority vote ( ≥ five pathologists) in 90.9% (720/792). Considerable discordance is observed for non-invasive melanomas (complete agreement in only 10/73 cases). The expert panel disagrees with the local pathologists’ and dermatologists’ diagnoses in 14.9% and 33.5% of cases, respectively. This variability highlights the diagnostic challenges of early-stage melanomas and the need to reconsider how ground truth is established in routine care and AI research. Including at least two pathologists or virtual panels may contribute to more consistent diagnostic results.https://doi.org/10.1038/s41467-025-56160-x |
| spellingShingle | Sarah Haggenmüller Christoph Wies Julia Abels Jana T. Winterstein Lukas Heinlein Carina Nogueira Garcia Jochen S. Utikal Sebastian A. Wohlfeil Friedegund Meier Sarah Hobelsberger Frank F. Gellrich Mildred Sergon Axel Hauschild Lars E. French Lucie Heinzerling Justin G. Schlager Kamran Ghoreschi Max Schlaak Franz J. Hilke Gabriela Poch Sören Korsing Cosimo Sarfert Carola Berking Markus V. Heppt Michael Erdmann Sebastian Haferkamp Konstantin Drexler Dirk Schadendorf Wiebke Sondermann Matthias Goebeler Bastian Schilling Jakob Nikolas Kather Stefan Fröhling Mar Llamas-Velasco Luis C. Requena Gerardo Ferrara Maite Fernandez-Figueras Sylvie Fraitag Cornelia S. L. Müller Hans Starz Heinz Kutzner Raymond Barnhill Richard Carr Kenneth S. Resnik Stephan Alexander Braun Tim Holland-Letz Titus J. Brinker Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors Nature Communications |
| title | Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors |
| title_full | Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors |
| title_fullStr | Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors |
| title_full_unstemmed | Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors |
| title_short | Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors |
| title_sort | discordance accuracy and reproducibility study of pathologists diagnosis of melanoma and melanocytic tumors |
| url | https://doi.org/10.1038/s41467-025-56160-x |
| work_keys_str_mv | AT sarahhaggenmuller discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT christophwies discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT juliaabels discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT janatwinterstein discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT lukasheinlein discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT carinanogueiragarcia discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT jochensutikal discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT sebastianawohlfeil discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT friedegundmeier discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT sarahhobelsberger discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT frankfgellrich discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT mildredsergon discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT axelhauschild discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT larsefrench discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT lucieheinzerling discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT justingschlager discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT kamranghoreschi discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT maxschlaak discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT franzjhilke discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT gabrielapoch discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT sorenkorsing discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT cosimosarfert discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT carolaberking discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT markusvheppt discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT michaelerdmann discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT sebastianhaferkamp discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT konstantindrexler discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT dirkschadendorf discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT wiebkesondermann discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT matthiasgoebeler discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT bastianschilling discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT jakobnikolaskather discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT stefanfrohling discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT marllamasvelasco discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT luiscrequena discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT gerardoferrara discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT maitefernandezfigueras discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT sylviefraitag discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT corneliaslmuller discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT hansstarz discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT heinzkutzner discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT raymondbarnhill discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT richardcarr discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT kennethsresnik discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT stephanalexanderbraun discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT timhollandletz discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors AT titusjbrinker discordanceaccuracyandreproducibilitystudyofpathologistsdiagnosisofmelanomaandmelanocytictumors |