A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced Cardiotoxicity
Chemotherapy-induced cardiotoxicity presents a major risk to cancer patients, often leading to severe cardiac complications such as heart failure, myocardial infarction, and arrhythmias. Early detection is crucial for preventing long-term damage and improving patient outcomes, yet existing diagnosti...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10946882/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850152166217482240 |
|---|---|
| author | Ahmed Bouatmane Abdelaziz Daaif Abdelmajid Bousselham Bouchra Bouihi Omar Bouattane |
| author_facet | Ahmed Bouatmane Abdelaziz Daaif Abdelmajid Bousselham Bouchra Bouihi Omar Bouattane |
| author_sort | Ahmed Bouatmane |
| collection | DOAJ |
| description | Chemotherapy-induced cardiotoxicity presents a major risk to cancer patients, often leading to severe cardiac complications such as heart failure, myocardial infarction, and arrhythmias. Early detection is crucial for preventing long-term damage and improving patient outcomes, yet existing diagnostic methods struggle to effectively capture the complexity of multimodal medical data and often lack interpretability. In this study, we propose an innovative approach that integrates multimodal deep learning with Explainable AI (XAI) techniques to enhance early cardiotoxicity detection. Our model combines clinical data (e.g., age and cardiovascular metrics) with Tissue Doppler Imaging (TDI), a functional imaging technique that captures myocardial velocity during the cardiac cycle. To overcome data limitations, we employed Conditional Generative Adversarial Networks (cGANs) and Conditional Tabular Generative Adversarial Networks (CTGANs) to augment the dataset, improving its diversity and balance for better model training. We developed three architectures that integrate Convolutional Neural Networks (CNNs) for feature extraction from TDI images with Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Transformer models to capture temporal dependencies and enhance prediction accuracy. Additionally, we incorporated SHapley Additive Explanations (SHAP) to interpret the contribution of input features, increasing model transparency and clinical applicability. Our Transformer-based model achieved the highest accuracy of 96%, outperforming the GRU (94%) and LSTM (89%) models, significantly surpassing traditional approaches. These findings highlight the potential of transformer-based architectures in multimodal deep learning for precise cardiotoxicity prediction, supporting early intervention and personalized treatment strategies while improving interpretability through XAI techniques such as SHAP. |
| format | Article |
| id | doaj-art-bdb1a74ce3964123a9add7da4b02bc32 |
| institution | OA Journals |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-bdb1a74ce3964123a9add7da4b02bc322025-08-20T02:26:03ZengIEEEIEEE Access2169-35362025-01-0113575685758810.1109/ACCESS.2025.355670010946882A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced CardiotoxicityAhmed Bouatmane0https://orcid.org/0000-0002-1732-551XAbdelaziz Daaif1Abdelmajid Bousselham2https://orcid.org/0000-0001-5458-2294Bouchra Bouihi3https://orcid.org/0000-0002-1652-8470Omar Bouattane42IACS Laboratory, ENSET Mohammedia, University of Hassan II, Casablanca, Morocco2IACS Laboratory, ENSET Mohammedia, University of Hassan II, Casablanca, Morocco2IACS Laboratory, ENSET Mohammedia, University of Hassan II, Casablanca, Morocco2IACS Laboratory, ENSET Mohammedia, University of Hassan II, Casablanca, MoroccoIESI Laboratory, ENSET Mohammedia, University of Hassan II, Casablanca, MoroccoChemotherapy-induced cardiotoxicity presents a major risk to cancer patients, often leading to severe cardiac complications such as heart failure, myocardial infarction, and arrhythmias. Early detection is crucial for preventing long-term damage and improving patient outcomes, yet existing diagnostic methods struggle to effectively capture the complexity of multimodal medical data and often lack interpretability. In this study, we propose an innovative approach that integrates multimodal deep learning with Explainable AI (XAI) techniques to enhance early cardiotoxicity detection. Our model combines clinical data (e.g., age and cardiovascular metrics) with Tissue Doppler Imaging (TDI), a functional imaging technique that captures myocardial velocity during the cardiac cycle. To overcome data limitations, we employed Conditional Generative Adversarial Networks (cGANs) and Conditional Tabular Generative Adversarial Networks (CTGANs) to augment the dataset, improving its diversity and balance for better model training. We developed three architectures that integrate Convolutional Neural Networks (CNNs) for feature extraction from TDI images with Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Transformer models to capture temporal dependencies and enhance prediction accuracy. Additionally, we incorporated SHapley Additive Explanations (SHAP) to interpret the contribution of input features, increasing model transparency and clinical applicability. Our Transformer-based model achieved the highest accuracy of 96%, outperforming the GRU (94%) and LSTM (89%) models, significantly surpassing traditional approaches. These findings highlight the potential of transformer-based architectures in multimodal deep learning for precise cardiotoxicity prediction, supporting early intervention and personalized treatment strategies while improving interpretability through XAI techniques such as SHAP.https://ieeexplore.ieee.org/document/10946882/Multimodal deep learningcardiotoxicitychemotherapyconvolutional neural networksrecurrent neural networkstransformer |
| spellingShingle | Ahmed Bouatmane Abdelaziz Daaif Abdelmajid Bousselham Bouchra Bouihi Omar Bouattane A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced Cardiotoxicity IEEE Access Multimodal deep learning cardiotoxicity chemotherapy convolutional neural networks recurrent neural networks transformer |
| title | A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced Cardiotoxicity |
| title_full | A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced Cardiotoxicity |
| title_fullStr | A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced Cardiotoxicity |
| title_full_unstemmed | A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced Cardiotoxicity |
| title_short | A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced Cardiotoxicity |
| title_sort | multimodal deep learning model integrating cnn and transformer for predicting chemotherapy induced cardiotoxicity |
| topic | Multimodal deep learning cardiotoxicity chemotherapy convolutional neural networks recurrent neural networks transformer |
| url | https://ieeexplore.ieee.org/document/10946882/ |
| work_keys_str_mv | AT ahmedbouatmane amultimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT abdelazizdaaif amultimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT abdelmajidbousselham amultimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT bouchrabouihi amultimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT omarbouattane amultimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT ahmedbouatmane multimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT abdelazizdaaif multimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT abdelmajidbousselham multimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT bouchrabouihi multimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity AT omarbouattane multimodaldeeplearningmodelintegratingcnnandtransformerforpredictingchemotherapyinducedcardiotoxicity |