Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion
In recent years, physiological signals such as electroencephalograhpy (EEG) have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However, the single-modal EEG signal has the problem of incomplete emotional information representation...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
POSTS&TELECOM PRESS Co., LTD
2021-03-01
|
| Series: | 智能科学与技术学报 |
| Subjects: | |
| Online Access: | http://www.cjist.com.cn/thesisDetails#10.11959/j.issn.2096-6652.202108 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850194604957106176 |
|---|---|
| author | Wenfen LING Sihan CHEN Yong PENG Wanzeng KONG |
| author_facet | Wenfen LING Sihan CHEN Yong PENG Wanzeng KONG |
| author_sort | Wenfen LING |
| collection | DOAJ |
| description | In recent years, physiological signals such as electroencephalograhpy (EEG) have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However, the single-modal EEG signal has the problem of incomplete emotional information representation, and the multi-modal physiological signal has the problem of insufficient emotional information interaction.Therefore, a 3D hierarchical convolutional fusion model was proposed, which aimed to fully explore multi-modal interaction relationships and more accurately describe emotional information.The method first extracted the primary emotional representation information of EEG , electro-oculogram (EOG) and electromyography (EMG) by depthwise separable convolution network, and then performed 3D convolution fusion operation on the obtained multi-modal primary emotional representation information to realize the pairwise mode local interactions between states and global interactions among all modalities, so as to obtain multi-modal fusion representations containing emotional characteristics of different physiological signals.The results show that the accuracy in the valence and arousal of the two-class and four-class tasks on DEAP dataset are both 98% by the proposed model. |
| format | Article |
| id | doaj-art-44af144182a8464b953ca31f6a51ea84 |
| institution | OA Journals |
| issn | 2096-6652 |
| language | zho |
| publishDate | 2021-03-01 |
| publisher | POSTS&TELECOM PRESS Co., LTD |
| record_format | Article |
| series | 智能科学与技术学报 |
| spelling | doaj-art-44af144182a8464b953ca31f6a51ea842025-08-20T02:13:58ZzhoPOSTS&TELECOM PRESS Co., LTD智能科学与技术学报2096-66522021-03-013768459639363Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusionWenfen LINGSihan CHENYong PENGWanzeng KONGIn recent years, physiological signals such as electroencephalograhpy (EEG) have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However, the single-modal EEG signal has the problem of incomplete emotional information representation, and the multi-modal physiological signal has the problem of insufficient emotional information interaction.Therefore, a 3D hierarchical convolutional fusion model was proposed, which aimed to fully explore multi-modal interaction relationships and more accurately describe emotional information.The method first extracted the primary emotional representation information of EEG , electro-oculogram (EOG) and electromyography (EMG) by depthwise separable convolution network, and then performed 3D convolution fusion operation on the obtained multi-modal primary emotional representation information to realize the pairwise mode local interactions between states and global interactions among all modalities, so as to obtain multi-modal fusion representations containing emotional characteristics of different physiological signals.The results show that the accuracy in the valence and arousal of the two-class and four-class tasks on DEAP dataset are both 98% by the proposed model.http://www.cjist.com.cn/thesisDetails#10.11959/j.issn.2096-6652.202108physiological signal;emotion recognition;3D hierarchical convolutional;multi-modal interaction |
| spellingShingle | Wenfen LING Sihan CHEN Yong PENG Wanzeng KONG Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion 智能科学与技术学报 physiological signal;emotion recognition;3D hierarchical convolutional;multi-modal interaction |
| title | Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion |
| title_full | Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion |
| title_fullStr | Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion |
| title_full_unstemmed | Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion |
| title_short | Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion |
| title_sort | multi modal physiological signal emotion recognition based on 3d hierarchical convolution fusion |
| topic | physiological signal;emotion recognition;3D hierarchical convolutional;multi-modal interaction |
| url | http://www.cjist.com.cn/thesisDetails#10.11959/j.issn.2096-6652.202108 |
| work_keys_str_mv | AT wenfenling multimodalphysiologicalsignalemotionrecognitionbasedon3dhierarchicalconvolutionfusion AT sihanchen multimodalphysiologicalsignalemotionrecognitionbasedon3dhierarchicalconvolutionfusion AT yongpeng multimodalphysiologicalsignalemotionrecognitionbasedon3dhierarchicalconvolutionfusion AT wanzengkong multimodalphysiologicalsignalemotionrecognitionbasedon3dhierarchicalconvolutionfusion |