Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability
Abstract Objective This study aims to develop an enhanced Transformer model for predicting mild cognitive impairment (MCI) using data from the China Health and Retirement Longitudinal Study (CHARLS), focusing on handling mixed data types and improving predictive accuracy. Methods The Transformer int...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-02-01
|
| Series: | BMC Public Health |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12889-025-21762-z |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823861601088831488 |
|---|---|
| author | Kai Ma Junzhi Zhang Xinhang Huang Mengyang Wang |
| author_facet | Kai Ma Junzhi Zhang Xinhang Huang Mengyang Wang |
| author_sort | Kai Ma |
| collection | DOAJ |
| description | Abstract Objective This study aims to develop an enhanced Transformer model for predicting mild cognitive impairment (MCI) using data from the China Health and Retirement Longitudinal Study (CHARLS), focusing on handling mixed data types and improving predictive accuracy. Methods The Transformer integrates categorical (integer-encoded) and continuous (floating-point) data, using multi-head attention with four heads to capture complex relationships. Preprocessing involved separate embedding layers for categorical data and feed-forward networks for continuous data. The model was compared with SVM and XGBoost, trained for 150 epochs with RMSProp and a cosine annealing scheduler. Key metrics included accuracy, Mean Absolute Error (MAE) tolerance, and training loss. An attention heatmap was generated to visualize feature importance. Results The Transformer outperformed SVM and XGBoost, achieving over 90% accuracy at an MAE tolerance of 3.5. The model showed rapid convergence, with loss stabilizing within 20 epochs. The attention heatmap highlighted key features, confirming the effectiveness of the multi-head attention mechanism in identifying relevant variables. Conclusion The enhanced Transformer model offers superior accuracy and efficiency in predicting cognitive decline compared to traditional models. Its capacity to process both continuous and categorical data and its interpretability through attention mechanisms make it a promising tool for early detection of neurodegenerative diseases, potentially improving clinical decision-making and interventions. |
| format | Article |
| id | doaj-art-e027d61aa3d5440290718a44e3d7a0d6 |
| institution | Kabale University |
| issn | 1471-2458 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Public Health |
| spelling | doaj-art-e027d61aa3d5440290718a44e3d7a0d62025-02-09T12:57:45ZengBMCBMC Public Health1471-24582025-02-0125111210.1186/s12889-025-21762-zLeveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretabilityKai Ma0Junzhi Zhang1Xinhang Huang2Mengyang Wang3College of Culture and Health Communication, Tianjin University of Traditional Chinese MedicineSchool of Public Health and Health Sciences, Tianjin University of Traditional Chinese MedicineCollege of Engineering, China Agricultural UniversitySchool of Public Health and Health Sciences, Tianjin University of Traditional Chinese MedicineAbstract Objective This study aims to develop an enhanced Transformer model for predicting mild cognitive impairment (MCI) using data from the China Health and Retirement Longitudinal Study (CHARLS), focusing on handling mixed data types and improving predictive accuracy. Methods The Transformer integrates categorical (integer-encoded) and continuous (floating-point) data, using multi-head attention with four heads to capture complex relationships. Preprocessing involved separate embedding layers for categorical data and feed-forward networks for continuous data. The model was compared with SVM and XGBoost, trained for 150 epochs with RMSProp and a cosine annealing scheduler. Key metrics included accuracy, Mean Absolute Error (MAE) tolerance, and training loss. An attention heatmap was generated to visualize feature importance. Results The Transformer outperformed SVM and XGBoost, achieving over 90% accuracy at an MAE tolerance of 3.5. The model showed rapid convergence, with loss stabilizing within 20 epochs. The attention heatmap highlighted key features, confirming the effectiveness of the multi-head attention mechanism in identifying relevant variables. Conclusion The enhanced Transformer model offers superior accuracy and efficiency in predicting cognitive decline compared to traditional models. Its capacity to process both continuous and categorical data and its interpretability through attention mechanisms make it a promising tool for early detection of neurodegenerative diseases, potentially improving clinical decision-making and interventions.https://doi.org/10.1186/s12889-025-21762-zMild cognitive impairment (MCI)Alzheimer’s disease (AD)Transformer modelMachine learningNeurodegenerative disease prediction |
| spellingShingle | Kai Ma Junzhi Zhang Xinhang Huang Mengyang Wang Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability BMC Public Health Mild cognitive impairment (MCI) Alzheimer’s disease (AD) Transformer model Machine learning Neurodegenerative disease prediction |
| title | Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability |
| title_full | Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability |
| title_fullStr | Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability |
| title_full_unstemmed | Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability |
| title_short | Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability |
| title_sort | leveraging transformer models to predict cognitive impairment accuracy efficiency and interpretability |
| topic | Mild cognitive impairment (MCI) Alzheimer’s disease (AD) Transformer model Machine learning Neurodegenerative disease prediction |
| url | https://doi.org/10.1186/s12889-025-21762-z |
| work_keys_str_mv | AT kaima leveragingtransformermodelstopredictcognitiveimpairmentaccuracyefficiencyandinterpretability AT junzhizhang leveragingtransformermodelstopredictcognitiveimpairmentaccuracyefficiencyandinterpretability AT xinhanghuang leveragingtransformermodelstopredictcognitiveimpairmentaccuracyefficiencyandinterpretability AT mengyangwang leveragingtransformermodelstopredictcognitiveimpairmentaccuracyefficiencyandinterpretability |