Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning
Abstract Multimodal sentiment analysis (MSA) aims to use a variety of sensors to obtain and process information to predict the intensity and polarity of human emotions. The main challenges faced by current multi-modal sentiment analysis include: how the model extracts emotional information in a sing...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-01-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-025-85859-6 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832594866414026752 |
|---|---|
| author | Yujian Cai Xingguang Li Yingyu Zhang Jinsong Li Fazheng Zhu Lin Rao |
| author_facet | Yujian Cai Xingguang Li Yingyu Zhang Jinsong Li Fazheng Zhu Lin Rao |
| author_sort | Yujian Cai |
| collection | DOAJ |
| description | Abstract Multimodal sentiment analysis (MSA) aims to use a variety of sensors to obtain and process information to predict the intensity and polarity of human emotions. The main challenges faced by current multi-modal sentiment analysis include: how the model extracts emotional information in a single modality and realizes the complementary transmission of multimodal information; how to output relatively stable predictions even when the sentiment embodied in a single modality is inconsistent with the multi-modal label; how can the model ensure high accuracy when a single modal information is incomplete or the feature extraction performance not good. Traditional methods do not take into account the interaction of unimodal contextual information and multi-modal information. They also ignore the independence and correlation of different modalities, which perform poorly when multimodal sentiment representations are asymmetric. To address these issues, this paper first proposes unimodal feature extraction network (UFEN) to extract unimodal features with stronger representation capabilities; then introduces multi-task fusion network (MTFN) to improve the correlation and fusion effect between multiple modalities. Multilayer feature extraction, attention mechanisms and Transformer are used in the model to mine potential relationships between features. Experimental results on MOSI, MOSEI and SIMS datasets show that the proposed method achieves better performance on multimodal sentiment analysis tasks compared with state-of-the-art baselines. |
| format | Article |
| id | doaj-art-e4257d1c49994b28bf4484d78f5ff0e8 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-e4257d1c49994b28bf4484d78f5ff0e82025-01-19T12:17:45ZengNature PortfolioScientific Reports2045-23222025-01-0115112110.1038/s41598-025-85859-6Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learningYujian Cai0Xingguang Li1Yingyu Zhang2Jinsong Li3Fazheng Zhu4Lin Rao5School of Electronic Information Engineering, Changchun University of Science and TechnologySchool of Electronic Information Engineering, Changchun University of Science and TechnologySchool of Electronic Information Engineering, Changchun University of Science and TechnologySchool of Electronic Information Engineering, Changchun University of Science and TechnologySchool of Electronic Information Engineering, Changchun University of Science and TechnologySchool of Electronic Information Engineering, Changchun University of Science and TechnologyAbstract Multimodal sentiment analysis (MSA) aims to use a variety of sensors to obtain and process information to predict the intensity and polarity of human emotions. The main challenges faced by current multi-modal sentiment analysis include: how the model extracts emotional information in a single modality and realizes the complementary transmission of multimodal information; how to output relatively stable predictions even when the sentiment embodied in a single modality is inconsistent with the multi-modal label; how can the model ensure high accuracy when a single modal information is incomplete or the feature extraction performance not good. Traditional methods do not take into account the interaction of unimodal contextual information and multi-modal information. They also ignore the independence and correlation of different modalities, which perform poorly when multimodal sentiment representations are asymmetric. To address these issues, this paper first proposes unimodal feature extraction network (UFEN) to extract unimodal features with stronger representation capabilities; then introduces multi-task fusion network (MTFN) to improve the correlation and fusion effect between multiple modalities. Multilayer feature extraction, attention mechanisms and Transformer are used in the model to mine potential relationships between features. Experimental results on MOSI, MOSEI and SIMS datasets show that the proposed method achieves better performance on multimodal sentiment analysis tasks compared with state-of-the-art baselines.https://doi.org/10.1038/s41598-025-85859-6 |
| spellingShingle | Yujian Cai Xingguang Li Yingyu Zhang Jinsong Li Fazheng Zhu Lin Rao Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning Scientific Reports |
| title | Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning |
| title_full | Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning |
| title_fullStr | Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning |
| title_full_unstemmed | Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning |
| title_short | Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning |
| title_sort | multimodal sentiment analysis based on multi layer feature fusion and multi task learning |
| url | https://doi.org/10.1038/s41598-025-85859-6 |
| work_keys_str_mv | AT yujiancai multimodalsentimentanalysisbasedonmultilayerfeaturefusionandmultitasklearning AT xingguangli multimodalsentimentanalysisbasedonmultilayerfeaturefusionandmultitasklearning AT yingyuzhang multimodalsentimentanalysisbasedonmultilayerfeaturefusionandmultitasklearning AT jinsongli multimodalsentimentanalysisbasedonmultilayerfeaturefusionandmultitasklearning AT fazhengzhu multimodalsentimentanalysisbasedonmultilayerfeaturefusionandmultitasklearning AT linrao multimodalsentimentanalysisbasedonmultilayerfeaturefusionandmultitasklearning |