Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks
Most studies indicate that bimodal emotion recognition systems are more objective and accurate. However, many of these systems depend on physiological signals that require direct measurement, which introduces certain limitations. This study aims to develop a new bimodal emotion recognition system ba...
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/11006048/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850150863317762048 |
|---|---|
| author | Sayat Orynbassar Duygun Erol Barkana Evan Yershov Madiyar Nurgaliyev Ahmet Saymbetov Batyrbek Zholamanov Gulbakhar Dosymbetova Ainur Kapparova Nursultan Koshkarbay Nurzhigit Kuttybay Askhat Bolatbek Kymbat Kopbay Dinara Almen |
| author_facet | Sayat Orynbassar Duygun Erol Barkana Evan Yershov Madiyar Nurgaliyev Ahmet Saymbetov Batyrbek Zholamanov Gulbakhar Dosymbetova Ainur Kapparova Nursultan Koshkarbay Nurzhigit Kuttybay Askhat Bolatbek Kymbat Kopbay Dinara Almen |
| author_sort | Sayat Orynbassar |
| collection | DOAJ |
| description | Most studies indicate that bimodal emotion recognition systems are more objective and accurate. However, many of these systems depend on physiological signals that require direct measurement, which introduces certain limitations. This study aims to develop a new bimodal emotion recognition system based on skin temperature (SKT) and heart rate variability (HRV) using hybrid neural networks. Notably, these physiological signals can be measured remotely, addressing the limitations of direct measurement methods. The integration of these modalities enables the model to effectively utilize both spatial and temporal features for robust emotion classification. The hybrid neural network, combining a convolutional neural network and a gated recurrent unit (CNN+GRU), was trained on experimental SKT and HRV data collected from individuals experiencing basic emotions such as anger, disgust, fear, happiness, sadness, and surprise. The trained model achieved an accuracy of 95.58%, outperforming existing approaches that use only a single data modality. Confusion matrix analysis demonstrated high accuracy in recognizing most basic emotions. The results confirm the effectiveness of the proposed approach in combining physiological and visual signals for improved emotion recognition. |
| format | Article |
| id | doaj-art-d8f4f18356f84c8a8735029af5d3e14e |
| institution | OA Journals |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-d8f4f18356f84c8a8735029af5d3e14e2025-08-20T02:26:26ZengIEEEIEEE Access2169-35362025-01-0113882158822910.1109/ACCESS.2025.357078511006048Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural NetworksSayat Orynbassar0https://orcid.org/0009-0001-9124-2560Duygun Erol Barkana1https://orcid.org/0000-0002-8929-0459Evan Yershov2Madiyar Nurgaliyev3https://orcid.org/0000-0002-6795-5384Ahmet Saymbetov4https://orcid.org/0000-0003-3442-8550Batyrbek Zholamanov5Gulbakhar Dosymbetova6Ainur Kapparova7Nursultan Koshkarbay8https://orcid.org/0009-0004-2334-3249Nurzhigit Kuttybay9https://orcid.org/0000-0002-5723-6642Askhat Bolatbek10https://orcid.org/0009-0004-7613-5507Kymbat Kopbay11Dinara Almen12https://orcid.org/0009-0000-8527-4921Faculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanDepartment of Electrical and Electronics Engineering, Faculty of Engineering, Yeditepe Üniversitesi, İstanbul, TürkiyeFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanFaculty of Physics and Technology, Al-Farabi Kazakh National University, Almaty, KazakhstanMost studies indicate that bimodal emotion recognition systems are more objective and accurate. However, many of these systems depend on physiological signals that require direct measurement, which introduces certain limitations. This study aims to develop a new bimodal emotion recognition system based on skin temperature (SKT) and heart rate variability (HRV) using hybrid neural networks. Notably, these physiological signals can be measured remotely, addressing the limitations of direct measurement methods. The integration of these modalities enables the model to effectively utilize both spatial and temporal features for robust emotion classification. The hybrid neural network, combining a convolutional neural network and a gated recurrent unit (CNN+GRU), was trained on experimental SKT and HRV data collected from individuals experiencing basic emotions such as anger, disgust, fear, happiness, sadness, and surprise. The trained model achieved an accuracy of 95.58%, outperforming existing approaches that use only a single data modality. Confusion matrix analysis demonstrated high accuracy in recognizing most basic emotions. The results confirm the effectiveness of the proposed approach in combining physiological and visual signals for improved emotion recognition.https://ieeexplore.ieee.org/document/11006048/Emotion recognitionthermographic imagesheart rate variabilityCNNRNN |
| spellingShingle | Sayat Orynbassar Duygun Erol Barkana Evan Yershov Madiyar Nurgaliyev Ahmet Saymbetov Batyrbek Zholamanov Gulbakhar Dosymbetova Ainur Kapparova Nursultan Koshkarbay Nurzhigit Kuttybay Askhat Bolatbek Kymbat Kopbay Dinara Almen Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks IEEE Access Emotion recognition thermographic images heart rate variability CNN RNN |
| title | Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks |
| title_full | Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks |
| title_fullStr | Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks |
| title_full_unstemmed | Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks |
| title_short | Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks |
| title_sort | development of bimodal emotion recognition system based on skin temperature and heart rate variability using hybrid neural networks |
| topic | Emotion recognition thermographic images heart rate variability CNN RNN |
| url | https://ieeexplore.ieee.org/document/11006048/ |
| work_keys_str_mv | AT sayatorynbassar developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT duygunerolbarkana developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT evanyershov developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT madiyarnurgaliyev developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT ahmetsaymbetov developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT batyrbekzholamanov developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT gulbakhardosymbetova developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT ainurkapparova developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT nursultankoshkarbay developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT nurzhigitkuttybay developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT askhatbolatbek developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT kymbatkopbay developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks AT dinaraalmen developmentofbimodalemotionrecognitionsystembasedonskintemperatureandheartratevariabilityusinghybridneuralnetworks |