Data splitting to avoid information leakage with DataSAIL
Abstract Information leakage is an increasingly important topic in machine learning research for biomedical applications. When information leakage happens during a model’s training, it risks memorizing the training data instead of learning generalizable properties. This can lead to inflated performa...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-04-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-025-58606-8 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850145381249187840 |
|---|---|
| author | Roman Joeres David B. Blumenthal Olga V. Kalinina |
| author_facet | Roman Joeres David B. Blumenthal Olga V. Kalinina |
| author_sort | Roman Joeres |
| collection | DOAJ |
| description | Abstract Information leakage is an increasingly important topic in machine learning research for biomedical applications. When information leakage happens during a model’s training, it risks memorizing the training data instead of learning generalizable properties. This can lead to inflated performance metrics that do not reflect the actual performance at inference time. We present DataSAIL, a versatile Python package to facilitate leakage-reduced data splitting to enable realistic evaluation of machine learning models for biological data that are intended to be applied in out-of-distribution scenarios. DataSAIL is based on formulating the problem to find leakage-reduced data splits as a combinatorial optimization problem. We prove that this problem is NP-hard and provide a scalable heuristic based on clustering and integer linear programming. Finally, we empirically demonstrate DataSAIL’s impact on evaluating biomedical machine learning models. |
| format | Article |
| id | doaj-art-a8603ca4033f49539d554f91b21eb5fe |
| institution | OA Journals |
| issn | 2041-1723 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj-art-a8603ca4033f49539d554f91b21eb5fe2025-08-20T02:28:07ZengNature PortfolioNature Communications2041-17232025-04-0116111110.1038/s41467-025-58606-8Data splitting to avoid information leakage with DataSAILRoman Joeres0David B. Blumenthal1Olga V. Kalinina2Helmholtz Institute for Pharmaceutical Research Saarland (HIPS), Helmholtz Centre for Infection Research (HZI)Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-NürnbergHelmholtz Institute for Pharmaceutical Research Saarland (HIPS), Helmholtz Centre for Infection Research (HZI)Abstract Information leakage is an increasingly important topic in machine learning research for biomedical applications. When information leakage happens during a model’s training, it risks memorizing the training data instead of learning generalizable properties. This can lead to inflated performance metrics that do not reflect the actual performance at inference time. We present DataSAIL, a versatile Python package to facilitate leakage-reduced data splitting to enable realistic evaluation of machine learning models for biological data that are intended to be applied in out-of-distribution scenarios. DataSAIL is based on formulating the problem to find leakage-reduced data splits as a combinatorial optimization problem. We prove that this problem is NP-hard and provide a scalable heuristic based on clustering and integer linear programming. Finally, we empirically demonstrate DataSAIL’s impact on evaluating biomedical machine learning models.https://doi.org/10.1038/s41467-025-58606-8 |
| spellingShingle | Roman Joeres David B. Blumenthal Olga V. Kalinina Data splitting to avoid information leakage with DataSAIL Nature Communications |
| title | Data splitting to avoid information leakage with DataSAIL |
| title_full | Data splitting to avoid information leakage with DataSAIL |
| title_fullStr | Data splitting to avoid information leakage with DataSAIL |
| title_full_unstemmed | Data splitting to avoid information leakage with DataSAIL |
| title_short | Data splitting to avoid information leakage with DataSAIL |
| title_sort | data splitting to avoid information leakage with datasail |
| url | https://doi.org/10.1038/s41467-025-58606-8 |
| work_keys_str_mv | AT romanjoeres datasplittingtoavoidinformationleakagewithdatasail AT davidbblumenthal datasplittingtoavoidinformationleakagewithdatasail AT olgavkalinina datasplittingtoavoidinformationleakagewithdatasail |