From prediction to practice: mitigating bias and data shift in machine-learning models for chemotherapy-induced organ dysfunction across unseen cancers
Objectives Routine monitoring of renal and hepatic function during chemotherapy ensures that treatment-related organ damage has not occurred and clearance of subsequent treatment is not hindered; however, frequency and timing are not optimal. Model bias and data heterogeneity concerns have hampered...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMJ Publishing Group
2024-08-01
|
| Series: | BMJ Oncology |
| Online Access: | https://bmjoncology.bmj.com/content/3/1/e000430.full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849321307144978432 |
|---|---|
| author | Heather Shaw Pinkie Chambers Matthew Watson Luke Steventon James Harmsworth King Angelo Ercia Noura Al Moubayed |
| author_facet | Heather Shaw Pinkie Chambers Matthew Watson Luke Steventon James Harmsworth King Angelo Ercia Noura Al Moubayed |
| author_sort | Heather Shaw |
| collection | DOAJ |
| description | Objectives Routine monitoring of renal and hepatic function during chemotherapy ensures that treatment-related organ damage has not occurred and clearance of subsequent treatment is not hindered; however, frequency and timing are not optimal. Model bias and data heterogeneity concerns have hampered the ability of machine learning (ML) to be deployed into clinical practice. This study aims to develop models that could support individualised decisions on the timing of renal and hepatic monitoring while exploring the effect of data shift on model performance.Methods and analysis We used retrospective data from three UK hospitals to develop and validate ML models predicting unacceptable rises in creatinine/bilirubin post cycle 3 for patients undergoing treatment for the following cancers: breast, colorectal, lung, ovarian and diffuse large B-cell lymphoma.Results We extracted 3614 patients with no missing blood test data across cycles 1–6 of chemotherapy treatment. We improved on previous work by including predictions post cycle 3. Optimised for sensitivity, we achieve F2 scores of 0.7773 (bilirubin) and 0.6893 (creatinine) on unseen data. Performance is consistent on tumour types unseen during training (F2 bilirubin: 0.7423, F2 creatinine: 0.6820).Conclusion Our technique highlights the effectiveness of ML in clinical settings, demonstrating the potential to improve the delivery of care. Notably, our ML models can generalise to unseen tumour types. We propose gold-standard bias mitigation steps for ML models: evaluation on multisite data, thorough patient population analysis, and both formalised bias measures and model performance comparisons on patient subgroups. We demonstrate that data aggregation techniques have unintended consequences on model bias. |
| format | Article |
| id | doaj-art-3bf2d554d58242faba7aeaec4b18144c |
| institution | Kabale University |
| issn | 2752-7948 |
| language | English |
| publishDate | 2024-08-01 |
| publisher | BMJ Publishing Group |
| record_format | Article |
| series | BMJ Oncology |
| spelling | doaj-art-3bf2d554d58242faba7aeaec4b18144c2025-08-20T03:49:46ZengBMJ Publishing GroupBMJ Oncology2752-79482024-08-013110.1136/bmjonc-2024-000430From prediction to practice: mitigating bias and data shift in machine-learning models for chemotherapy-induced organ dysfunction across unseen cancersHeather Shaw0Pinkie Chambers1Matthew Watson2Luke Steventon3James Harmsworth King4Angelo Ercia5Noura Al Moubayed66University College London Hospital, London, UKUCLH-UCL Centre for Medicines Optimisation Research and Education, University College London NHS Foundation Trust, London, UKCancer Division, University College London Hospitals NHS Foundation Trust, London, UKRDPP, UCL School of Pharmacy, London, UKEvergreen Life Ltd, Manchester, UKEvergreen Life Ltd, Manchester, UKDepartment of Computer Science, Durham University, Durham, UKObjectives Routine monitoring of renal and hepatic function during chemotherapy ensures that treatment-related organ damage has not occurred and clearance of subsequent treatment is not hindered; however, frequency and timing are not optimal. Model bias and data heterogeneity concerns have hampered the ability of machine learning (ML) to be deployed into clinical practice. This study aims to develop models that could support individualised decisions on the timing of renal and hepatic monitoring while exploring the effect of data shift on model performance.Methods and analysis We used retrospective data from three UK hospitals to develop and validate ML models predicting unacceptable rises in creatinine/bilirubin post cycle 3 for patients undergoing treatment for the following cancers: breast, colorectal, lung, ovarian and diffuse large B-cell lymphoma.Results We extracted 3614 patients with no missing blood test data across cycles 1–6 of chemotherapy treatment. We improved on previous work by including predictions post cycle 3. Optimised for sensitivity, we achieve F2 scores of 0.7773 (bilirubin) and 0.6893 (creatinine) on unseen data. Performance is consistent on tumour types unseen during training (F2 bilirubin: 0.7423, F2 creatinine: 0.6820).Conclusion Our technique highlights the effectiveness of ML in clinical settings, demonstrating the potential to improve the delivery of care. Notably, our ML models can generalise to unseen tumour types. We propose gold-standard bias mitigation steps for ML models: evaluation on multisite data, thorough patient population analysis, and both formalised bias measures and model performance comparisons on patient subgroups. We demonstrate that data aggregation techniques have unintended consequences on model bias.https://bmjoncology.bmj.com/content/3/1/e000430.full |
| spellingShingle | Heather Shaw Pinkie Chambers Matthew Watson Luke Steventon James Harmsworth King Angelo Ercia Noura Al Moubayed From prediction to practice: mitigating bias and data shift in machine-learning models for chemotherapy-induced organ dysfunction across unseen cancers BMJ Oncology |
| title | From prediction to practice: mitigating bias and data shift in machine-learning models for chemotherapy-induced organ dysfunction across unseen cancers |
| title_full | From prediction to practice: mitigating bias and data shift in machine-learning models for chemotherapy-induced organ dysfunction across unseen cancers |
| title_fullStr | From prediction to practice: mitigating bias and data shift in machine-learning models for chemotherapy-induced organ dysfunction across unseen cancers |
| title_full_unstemmed | From prediction to practice: mitigating bias and data shift in machine-learning models for chemotherapy-induced organ dysfunction across unseen cancers |
| title_short | From prediction to practice: mitigating bias and data shift in machine-learning models for chemotherapy-induced organ dysfunction across unseen cancers |
| title_sort | from prediction to practice mitigating bias and data shift in machine learning models for chemotherapy induced organ dysfunction across unseen cancers |
| url | https://bmjoncology.bmj.com/content/3/1/e000430.full |
| work_keys_str_mv | AT heathershaw frompredictiontopracticemitigatingbiasanddatashiftinmachinelearningmodelsforchemotherapyinducedorgandysfunctionacrossunseencancers AT pinkiechambers frompredictiontopracticemitigatingbiasanddatashiftinmachinelearningmodelsforchemotherapyinducedorgandysfunctionacrossunseencancers AT matthewwatson frompredictiontopracticemitigatingbiasanddatashiftinmachinelearningmodelsforchemotherapyinducedorgandysfunctionacrossunseencancers AT lukesteventon frompredictiontopracticemitigatingbiasanddatashiftinmachinelearningmodelsforchemotherapyinducedorgandysfunctionacrossunseencancers AT jamesharmsworthking frompredictiontopracticemitigatingbiasanddatashiftinmachinelearningmodelsforchemotherapyinducedorgandysfunctionacrossunseencancers AT angeloercia frompredictiontopracticemitigatingbiasanddatashiftinmachinelearningmodelsforchemotherapyinducedorgandysfunctionacrossunseencancers AT nouraalmoubayed frompredictiontopracticemitigatingbiasanddatashiftinmachinelearningmodelsforchemotherapyinducedorgandysfunctionacrossunseencancers |