Early and Late Fusion for Multimodal Aggression Prediction in Dementia Patients: A Comparative Analysis
Aggression in patients with dementia poses significant caregiving and clinical issues. In this work, fusion approaches—Early Fusion and Late Fusion—were compared to classify aggression using audio and visual signals. Early Fusion integrates the extracted features of the two modalities into one datas...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-05-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/11/5823 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849330693633474560 |
|---|---|
| author | Ioannis Galanakis Rigas Filippos Soldatos Nikitas Karanikolas Athanasios Voulodimos Ioannis Voyiatzis Maria Samarakou |
| author_facet | Ioannis Galanakis Rigas Filippos Soldatos Nikitas Karanikolas Athanasios Voulodimos Ioannis Voyiatzis Maria Samarakou |
| author_sort | Ioannis Galanakis |
| collection | DOAJ |
| description | Aggression in patients with dementia poses significant caregiving and clinical issues. In this work, fusion approaches—Early Fusion and Late Fusion—were compared to classify aggression using audio and visual signals. Early Fusion integrates the extracted features of the two modalities into one dataset before classification, while Late Fusion integrates the prediction probabilities of standalone audio and visual classifiers with a meta-classifier. Both models were tested using a Random Forest classifier with five-fold cross-validation, and the performance was compared on accuracy, precision, recall, F1-score, ROC-AUC, and inference time. The results showcase that Late Fusion is superior to Early Fusion in terms of accuracy (0.876 vs. 0.828), recall (0.914 vs. 0.818), F1-score (0.867 vs. 0.835), and ROC-AUC score (0.970 vs. 0.922), proving more suitable for high-sensitivity use cases like healthcare and security. However, Early Fusion exhibited higher precision (0.852 vs. 0.824), indicating that in cases when false positives are a requirement, Early Fusion is preferable. Paired <i>t</i>-tests were applied for statistical comparison and indicate that precision alone is significantly different, with the advantage of Early Fusion. Late Fusion also performs slightly less in inference time, which makes it suitable for use in real-time systems. These findings provide significant information on multimodal fusion strategies and their applicability in the detection of aggressive behavior, which can contribute to the development of efficient monitoring systems for dementia care. |
| format | Article |
| id | doaj-art-c7d1f601202b4bdcb6827db774fcd1b1 |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-c7d1f601202b4bdcb6827db774fcd1b12025-08-20T03:46:50ZengMDPI AGApplied Sciences2076-34172025-05-011511582310.3390/app15115823Early and Late Fusion for Multimodal Aggression Prediction in Dementia Patients: A Comparative AnalysisIoannis Galanakis0Rigas Filippos Soldatos1Nikitas Karanikolas2Athanasios Voulodimos3Ioannis Voyiatzis4Maria Samarakou5Department of Informatics and Computing Engineering, University of West Attica, 12243 Athens, GreeceFirst Department of Psychiatry, Eginition Hospital, National and Kapodistrian University of Athens Medical School, 11528 Athens, GreeceDepartment of Informatics and Computing Engineering, University of West Attica, 12243 Athens, GreeceDepartment of School of Electrical & Computing Engineering, National Technical University of Athens, 15780 Athens, GreeceDepartment of Informatics and Computing Engineering, University of West Attica, 12243 Athens, GreeceDepartment of Informatics and Computing Engineering, University of West Attica, 12243 Athens, GreeceAggression in patients with dementia poses significant caregiving and clinical issues. In this work, fusion approaches—Early Fusion and Late Fusion—were compared to classify aggression using audio and visual signals. Early Fusion integrates the extracted features of the two modalities into one dataset before classification, while Late Fusion integrates the prediction probabilities of standalone audio and visual classifiers with a meta-classifier. Both models were tested using a Random Forest classifier with five-fold cross-validation, and the performance was compared on accuracy, precision, recall, F1-score, ROC-AUC, and inference time. The results showcase that Late Fusion is superior to Early Fusion in terms of accuracy (0.876 vs. 0.828), recall (0.914 vs. 0.818), F1-score (0.867 vs. 0.835), and ROC-AUC score (0.970 vs. 0.922), proving more suitable for high-sensitivity use cases like healthcare and security. However, Early Fusion exhibited higher precision (0.852 vs. 0.824), indicating that in cases when false positives are a requirement, Early Fusion is preferable. Paired <i>t</i>-tests were applied for statistical comparison and indicate that precision alone is significantly different, with the advantage of Early Fusion. Late Fusion also performs slightly less in inference time, which makes it suitable for use in real-time systems. These findings provide significant information on multimodal fusion strategies and their applicability in the detection of aggressive behavior, which can contribute to the development of efficient monitoring systems for dementia care.https://www.mdpi.com/2076-3417/15/11/5823machine learningmultimodal analysislate fusionearly fusioncomparative analysismeta-classifier |
| spellingShingle | Ioannis Galanakis Rigas Filippos Soldatos Nikitas Karanikolas Athanasios Voulodimos Ioannis Voyiatzis Maria Samarakou Early and Late Fusion for Multimodal Aggression Prediction in Dementia Patients: A Comparative Analysis Applied Sciences machine learning multimodal analysis late fusion early fusion comparative analysis meta-classifier |
| title | Early and Late Fusion for Multimodal Aggression Prediction in Dementia Patients: A Comparative Analysis |
| title_full | Early and Late Fusion for Multimodal Aggression Prediction in Dementia Patients: A Comparative Analysis |
| title_fullStr | Early and Late Fusion for Multimodal Aggression Prediction in Dementia Patients: A Comparative Analysis |
| title_full_unstemmed | Early and Late Fusion for Multimodal Aggression Prediction in Dementia Patients: A Comparative Analysis |
| title_short | Early and Late Fusion for Multimodal Aggression Prediction in Dementia Patients: A Comparative Analysis |
| title_sort | early and late fusion for multimodal aggression prediction in dementia patients a comparative analysis |
| topic | machine learning multimodal analysis late fusion early fusion comparative analysis meta-classifier |
| url | https://www.mdpi.com/2076-3417/15/11/5823 |
| work_keys_str_mv | AT ioannisgalanakis earlyandlatefusionformultimodalaggressionpredictionindementiapatientsacomparativeanalysis AT rigasfilippossoldatos earlyandlatefusionformultimodalaggressionpredictionindementiapatientsacomparativeanalysis AT nikitaskaranikolas earlyandlatefusionformultimodalaggressionpredictionindementiapatientsacomparativeanalysis AT athanasiosvoulodimos earlyandlatefusionformultimodalaggressionpredictionindementiapatientsacomparativeanalysis AT ioannisvoyiatzis earlyandlatefusionformultimodalaggressionpredictionindementiapatientsacomparativeanalysis AT mariasamarakou earlyandlatefusionformultimodalaggressionpredictionindementiapatientsacomparativeanalysis |