MMEmAsis: multimodal emotion and sentiment analysis
The paper presents a new multimodal approach to analyzing the psycho-emotional state of a person using nonlinear classifiers. The main modalities are the subject’s speech data and video data of facial expressions. Speech is digitized and transcribed using the Scribe library, and then mood cues are e...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Peoples’ Friendship University of Russia (RUDN University)
2024-12-01
|
| Series: | Discrete and Continuous Models and Applied Computational Science |
| Subjects: | |
| Online Access: | https://journals.rudn.ru/miph/article/viewFile/43666/24663 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | The paper presents a new multimodal approach to analyzing the psycho-emotional state of a person using nonlinear classifiers. The main modalities are the subject’s speech data and video data of facial expressions. Speech is digitized and transcribed using the Scribe library, and then mood cues are extracted using the Titanis sentiment analyzer from the FRC CSC RAS. For visual analysis, two different approaches were implemented: a pre-trained ResNet model for direct sentiment classification from facial expressions, and a deep learning model that integrates ResNet with a graph-based deep neural network for facial recognition. Both approaches have faced challenges related to environmental factors affecting the stability of results. The second approach demonstrated greater flexibility with adjustable classification vocabularies, which facilitated post-deployment calibration. Integration of text and visual data has significantly improved the accuracy and reliability of the analysis of a person’s psycho-emotional state |
|---|---|
| ISSN: | 2658-4670 2658-7149 |