Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks

Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks

Most studies indicate that bimodal emotion recognition systems are more objective and accurate. However, many of these systems depend on physiological signals that require direct measurement, which introduces certain limitations. This study aims to develop a new bimodal emotion recognition system ba...

Full description

Saved in:
Bibliographic Details
Main Authors: Sayat Orynbassar, Duygun Erol Barkana, Evan Yershov, Madiyar Nurgaliyev, Ahmet Saymbetov, Batyrbek Zholamanov, Gulbakhar Dosymbetova, Ainur Kapparova, Nursultan Koshkarbay, Nurzhigit Kuttybay, Askhat Bolatbek, Kymbat Kopbay, Dinara Almen
Format: Article
Language:English
Published: IEEE 2025-01-01
Series:IEEE Access
Subjects:
Emotion recognition
thermographic images
heart rate variability
CNN
RNN
Online Access:https://ieeexplore.ieee.org/document/11006048/
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Most studies indicate that bimodal emotion recognition systems are more objective and accurate. However, many of these systems depend on physiological signals that require direct measurement, which introduces certain limitations. This study aims to develop a new bimodal emotion recognition system based on skin temperature (SKT) and heart rate variability (HRV) using hybrid neural networks. Notably, these physiological signals can be measured remotely, addressing the limitations of direct measurement methods. The integration of these modalities enables the model to effectively utilize both spatial and temporal features for robust emotion classification. The hybrid neural network, combining a convolutional neural network and a gated recurrent unit (CNN+GRU), was trained on experimental SKT and HRV data collected from individuals experiencing basic emotions such as anger, disgust, fear, happiness, sadness, and surprise. The trained model achieved an accuracy of 95.58%, outperforming existing approaches that use only a single data modality. Confusion matrix analysis demonstrated high accuracy in recognizing most basic emotions. The results confirm the effectiveness of the proposed approach in combining physiological and visual signals for improved emotion recognition.
ISSN:2169-3536

Similar Items