Open-radiomics: a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelines
Abstract Background As an important branch of machine learning pipelines in medical imaging, radiomics faces two major challenges namely reproducibility and accessibility. In this work, we introduce open-radiomics, a set of radiomics datasets along with a comprehensive radiomics pipeline based on ou...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-08-01
|
| Series: | BMC Medical Imaging |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12880-025-01855-2 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849332194167750656 |
|---|---|
| author | Khashayar Namdar Matthias W. Wagner Birgit B. Ertl-Wagner Farzad Khalvati |
| author_facet | Khashayar Namdar Matthias W. Wagner Birgit B. Ertl-Wagner Farzad Khalvati |
| author_sort | Khashayar Namdar |
| collection | DOAJ |
| description | Abstract Background As an important branch of machine learning pipelines in medical imaging, radiomics faces two major challenges namely reproducibility and accessibility. In this work, we introduce open-radiomics, a set of radiomics datasets along with a comprehensive radiomics pipeline based on our proposed technical protocol to investigate the effects of radiomics feature extraction on the reproducibility of the results. Methods We curated large-scale radiomics datasets based on three open-source datasets; BraTS 2020 for high-grade glioma (HGG) versus low-grade glioma (LGG) classification and survival analysis, BraTS 2023 for O6-methylguanine-DNA methyltransferase (MGMT) classification, and non-small cell lung cancer (NSCLC) survival analysis from the Cancer Imaging Archive (TCIA). We used the BraTS 2020 open-source Magnetic Resonance Imaging (MRI) dataset to demonstrate how our proposed technical protocol could be utilized in radiomics-based studies. The cohort includes 369 adult patients with brain tumors (76 LGG, and 293 HGG). Using PyRadiomics library for LGG vs. HGG classification, we created 288 radiomics datasets; the combinations of 4 MRI sequences, 3 binWidths, 6 image normalization methods, and 4 tumor subregions. We used Random Forest classifiers, and for each radiomics dataset, we repeated the training-validation-test (60%/20%/20%) experiment with different data splits and model random states 100 times (28,800 test results) and calculated the Area Under the Receiver Operating Characteristic Curve (AUROC). Results Unlike binWidth and image normalization, the tumor subregion and imaging sequence significantly affected performance of the models. T1 contrast-enhanced sequence and the union of Necrotic and the non-enhancing tumor core subregions resulted in the highest AUROCs (average test AUROC 0.951, 95% confidence interval of (0.949, 0.952)). Although several settings and data splits (28 out of 28800) yielded test AUROC of 1, they were irreproducible. Conclusions Our experiments demonstrate the sources of variability in radiomics pipelines (e.g., tumor subregion) can have a significant impact on the results, which may lead to superficial perfect performances that are irreproducible. Clinical trial number Not applicable. |
| format | Article |
| id | doaj-art-e6071b010297412fa8fc744a5507fec7 |
| institution | Kabale University |
| issn | 1471-2342 |
| language | English |
| publishDate | 2025-08-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Imaging |
| spelling | doaj-art-e6071b010297412fa8fc744a5507fec72025-08-20T03:46:17ZengBMCBMC Medical Imaging1471-23422025-08-0125111810.1186/s12880-025-01855-2Open-radiomics: a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelinesKhashayar Namdar0Matthias W. Wagner1Birgit B. Ertl-Wagner2Farzad Khalvati3Department of Diagnostic & Interventional Radiology, The Hospital for Sick Children (SickKids)Department of Diagnostic & Interventional Radiology, The Hospital for Sick Children (SickKids)Department of Diagnostic & Interventional Radiology, The Hospital for Sick Children (SickKids)Department of Diagnostic & Interventional Radiology, The Hospital for Sick Children (SickKids)Abstract Background As an important branch of machine learning pipelines in medical imaging, radiomics faces two major challenges namely reproducibility and accessibility. In this work, we introduce open-radiomics, a set of radiomics datasets along with a comprehensive radiomics pipeline based on our proposed technical protocol to investigate the effects of radiomics feature extraction on the reproducibility of the results. Methods We curated large-scale radiomics datasets based on three open-source datasets; BraTS 2020 for high-grade glioma (HGG) versus low-grade glioma (LGG) classification and survival analysis, BraTS 2023 for O6-methylguanine-DNA methyltransferase (MGMT) classification, and non-small cell lung cancer (NSCLC) survival analysis from the Cancer Imaging Archive (TCIA). We used the BraTS 2020 open-source Magnetic Resonance Imaging (MRI) dataset to demonstrate how our proposed technical protocol could be utilized in radiomics-based studies. The cohort includes 369 adult patients with brain tumors (76 LGG, and 293 HGG). Using PyRadiomics library for LGG vs. HGG classification, we created 288 radiomics datasets; the combinations of 4 MRI sequences, 3 binWidths, 6 image normalization methods, and 4 tumor subregions. We used Random Forest classifiers, and for each radiomics dataset, we repeated the training-validation-test (60%/20%/20%) experiment with different data splits and model random states 100 times (28,800 test results) and calculated the Area Under the Receiver Operating Characteristic Curve (AUROC). Results Unlike binWidth and image normalization, the tumor subregion and imaging sequence significantly affected performance of the models. T1 contrast-enhanced sequence and the union of Necrotic and the non-enhancing tumor core subregions resulted in the highest AUROCs (average test AUROC 0.951, 95% confidence interval of (0.949, 0.952)). Although several settings and data splits (28 out of 28800) yielded test AUROC of 1, they were irreproducible. Conclusions Our experiments demonstrate the sources of variability in radiomics pipelines (e.g., tumor subregion) can have a significant impact on the results, which may lead to superficial perfect performances that are irreproducible. Clinical trial number Not applicable.https://doi.org/10.1186/s12880-025-01855-2RadiomicsOpen-sourceDatasetBrain cancerReproducibility |
| spellingShingle | Khashayar Namdar Matthias W. Wagner Birgit B. Ertl-Wagner Farzad Khalvati Open-radiomics: a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelines BMC Medical Imaging Radiomics Open-source Dataset Brain cancer Reproducibility |
| title | Open-radiomics: a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelines |
| title_full | Open-radiomics: a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelines |
| title_fullStr | Open-radiomics: a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelines |
| title_full_unstemmed | Open-radiomics: a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelines |
| title_short | Open-radiomics: a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelines |
| title_sort | open radiomics a collection of standardized datasets and a technical protocol for reproducible radiomics machine learning pipelines |
| topic | Radiomics Open-source Dataset Brain cancer Reproducibility |
| url | https://doi.org/10.1186/s12880-025-01855-2 |
| work_keys_str_mv | AT khashayarnamdar openradiomicsacollectionofstandardizeddatasetsandatechnicalprotocolforreproducibleradiomicsmachinelearningpipelines AT matthiaswwagner openradiomicsacollectionofstandardizeddatasetsandatechnicalprotocolforreproducibleradiomicsmachinelearningpipelines AT birgitbertlwagner openradiomicsacollectionofstandardizeddatasetsandatechnicalprotocolforreproducibleradiomicsmachinelearningpipelines AT farzadkhalvati openradiomicsacollectionofstandardizeddatasetsandatechnicalprotocolforreproducibleradiomicsmachinelearningpipelines |