Attention-Based CNN Fusion Model for Emotion Recognition During Walking Using Discrete Wavelet Transform on EEG and Inertial Signals
Walking as a unique biometric tool conveys important information for emotion recognition. Individuals in different emotional states exhibit distinct walking patterns. For this purpose, this paper proposes a novel approach to recognizing emotion during walking using electroencephalogram (EEG) and ine...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Tsinghua University Press
2024-03-01
|
| Series: | Big Data Mining and Analytics |
| Subjects: | |
| Online Access: | https://www.sciopen.com/article/10.26599/BDMA.2023.9020018 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832544203297521664 |
|---|---|
| author | Yan Zhao Ming Guo Xiangyong Chen Jianqiang Sun Jianlong Qiu |
| author_facet | Yan Zhao Ming Guo Xiangyong Chen Jianqiang Sun Jianlong Qiu |
| author_sort | Yan Zhao |
| collection | DOAJ |
| description | Walking as a unique biometric tool conveys important information for emotion recognition. Individuals in different emotional states exhibit distinct walking patterns. For this purpose, this paper proposes a novel approach to recognizing emotion during walking using electroencephalogram (EEG) and inertial signals. Accurate recognition of emotion is achieved by training in an end-to-end deep learning fashion and taking into account multi-modal fusion. Subjects wear virtual reality head-mounted display (VR-HMD) equipment to immerse in strong emotions during walking. VR environment shows excellent imitation and experience ability, which plays an important role in awakening and changing emotions. In addition, the multi-modal signals acquired from EEG and inertial sensors are separately represented as virtual emotion images by discrete wavelet transform (DWT). These serve as input to the attention-based convolutional neural network (CNN) fusion model. The designed network structure is simple and lightweight while integrating the channel attention mechanism to extract and enhance features. To effectively improve the performance of the recognition system, the proposed decision fusion algorithm combines Critic method and majority voting strategy to determine the weight values that affect the final decision results. An investigation is made on the effect of diverse mother wavelet types and wavelet decomposition levels on model performance which indicates that the 2.2-order reverse biorthogonal (rbio2.2) wavelet with two-level decomposition has the best recognition performance. Comparative experiment results show that the proposed method outperforms other existing state-of-the-art works with an accuracy of 98.73%. |
| format | Article |
| id | doaj-art-6be306998f2a4298ab9850f655f69751 |
| institution | Kabale University |
| issn | 2096-0654 |
| language | English |
| publishDate | 2024-03-01 |
| publisher | Tsinghua University Press |
| record_format | Article |
| series | Big Data Mining and Analytics |
| spelling | doaj-art-6be306998f2a4298ab9850f655f697512025-02-03T10:49:41ZengTsinghua University PressBig Data Mining and Analytics2096-06542024-03-017118820410.26599/BDMA.2023.9020018Attention-Based CNN Fusion Model for Emotion Recognition During Walking Using Discrete Wavelet Transform on EEG and Inertial SignalsYan Zhao0Ming Guo1Xiangyong Chen2Jianqiang Sun3Jianlong Qiu4School of Communication Engineering, Hangzhou Dianzi University, Hangzhou 310018, China, and also with the School of Automation and Electrical Engineering, Linyi University, Linyi 276000, ChinaSchool of Automation and Electrical Engineering, Linyi University, Linyi 276000, ChinaSchool of Automation and Electrical Engineering, Linyi University, Linyi 276000, ChinaSchool of Automation and Electrical Engineering, Linyi University, Linyi 276000, ChinaSchool of Automation and Electrical Engineering, Linyi University, Linyi 276000, ChinaWalking as a unique biometric tool conveys important information for emotion recognition. Individuals in different emotional states exhibit distinct walking patterns. For this purpose, this paper proposes a novel approach to recognizing emotion during walking using electroencephalogram (EEG) and inertial signals. Accurate recognition of emotion is achieved by training in an end-to-end deep learning fashion and taking into account multi-modal fusion. Subjects wear virtual reality head-mounted display (VR-HMD) equipment to immerse in strong emotions during walking. VR environment shows excellent imitation and experience ability, which plays an important role in awakening and changing emotions. In addition, the multi-modal signals acquired from EEG and inertial sensors are separately represented as virtual emotion images by discrete wavelet transform (DWT). These serve as input to the attention-based convolutional neural network (CNN) fusion model. The designed network structure is simple and lightweight while integrating the channel attention mechanism to extract and enhance features. To effectively improve the performance of the recognition system, the proposed decision fusion algorithm combines Critic method and majority voting strategy to determine the weight values that affect the final decision results. An investigation is made on the effect of diverse mother wavelet types and wavelet decomposition levels on model performance which indicates that the 2.2-order reverse biorthogonal (rbio2.2) wavelet with two-level decomposition has the best recognition performance. Comparative experiment results show that the proposed method outperforms other existing state-of-the-art works with an accuracy of 98.73%.https://www.sciopen.com/article/10.26599/BDMA.2023.9020018walkingmulti-modal fusionvirtual realityemotion recognitiondiscrete wavelet transformattention mechanism |
| spellingShingle | Yan Zhao Ming Guo Xiangyong Chen Jianqiang Sun Jianlong Qiu Attention-Based CNN Fusion Model for Emotion Recognition During Walking Using Discrete Wavelet Transform on EEG and Inertial Signals Big Data Mining and Analytics walking multi-modal fusion virtual reality emotion recognition discrete wavelet transform attention mechanism |
| title | Attention-Based CNN Fusion Model for Emotion Recognition During Walking Using Discrete Wavelet Transform on EEG and Inertial Signals |
| title_full | Attention-Based CNN Fusion Model for Emotion Recognition During Walking Using Discrete Wavelet Transform on EEG and Inertial Signals |
| title_fullStr | Attention-Based CNN Fusion Model for Emotion Recognition During Walking Using Discrete Wavelet Transform on EEG and Inertial Signals |
| title_full_unstemmed | Attention-Based CNN Fusion Model for Emotion Recognition During Walking Using Discrete Wavelet Transform on EEG and Inertial Signals |
| title_short | Attention-Based CNN Fusion Model for Emotion Recognition During Walking Using Discrete Wavelet Transform on EEG and Inertial Signals |
| title_sort | attention based cnn fusion model for emotion recognition during walking using discrete wavelet transform on eeg and inertial signals |
| topic | walking multi-modal fusion virtual reality emotion recognition discrete wavelet transform attention mechanism |
| url | https://www.sciopen.com/article/10.26599/BDMA.2023.9020018 |
| work_keys_str_mv | AT yanzhao attentionbasedcnnfusionmodelforemotionrecognitionduringwalkingusingdiscretewavelettransformoneegandinertialsignals AT mingguo attentionbasedcnnfusionmodelforemotionrecognitionduringwalkingusingdiscretewavelettransformoneegandinertialsignals AT xiangyongchen attentionbasedcnnfusionmodelforemotionrecognitionduringwalkingusingdiscretewavelettransformoneegandinertialsignals AT jianqiangsun attentionbasedcnnfusionmodelforemotionrecognitionduringwalkingusingdiscretewavelettransformoneegandinertialsignals AT jianlongqiu attentionbasedcnnfusionmodelforemotionrecognitionduringwalkingusingdiscretewavelettransformoneegandinertialsignals |