Multi-HM: A Chinese Multimodal Dataset and Fusion Framework for Emotion Recognition in Human–Machine Dialogue Systems
Sentiment analysis is pivotal in advancing human–computer interaction (HCI) systems as it enables emotionally intelligent responses. While existing models show potential for HCI applications, current conversational datasets exhibit critical limitations in real-world deployment, particularly in captu...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-04-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/8/4509 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850183592092631040 |
|---|---|
| author | Yao Fu Qiong Liu Qing Song Pengzhou Zhang Gongdong Liao |
| author_facet | Yao Fu Qiong Liu Qing Song Pengzhou Zhang Gongdong Liao |
| author_sort | Yao Fu |
| collection | DOAJ |
| description | Sentiment analysis is pivotal in advancing human–computer interaction (HCI) systems as it enables emotionally intelligent responses. While existing models show potential for HCI applications, current conversational datasets exhibit critical limitations in real-world deployment, particularly in capturing domain-specific emotional dynamics and context-sensitive behavioral patterns—constraints that hinder semantic comprehension and adaptive capabilities in task-driven HCI scenarios. To address these gaps, we present Multi-HM, the first multimodal emotion recognition dataset explicitly designed for human–machine consultation systems. It contains 2000 professionally annotated dialogues across 10 major HCI domains. Our dataset employs a five-dimensional annotation framework that systematically integrates textual, vocal, and visual modalities while simulating authentic HCI workflows to encode pragmatic behavioral cues and mission-critical emotional trajectories. Experiments demonstrate that Multi-HM-trained models achieve state-of-the-art performance in recognizing task-oriented affective states. This resource establishes a crucial foundation for developing human-centric AI systems that dynamically adapt to users’ evolving emotional needs. |
| format | Article |
| id | doaj-art-160f6dedb1014c6d8d269dfd8b04e576 |
| institution | OA Journals |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-160f6dedb1014c6d8d269dfd8b04e5762025-08-20T02:17:19ZengMDPI AGApplied Sciences2076-34172025-04-01158450910.3390/app15084509Multi-HM: A Chinese Multimodal Dataset and Fusion Framework for Emotion Recognition in Human–Machine Dialogue SystemsYao Fu0Qiong Liu1Qing Song2Pengzhou Zhang3Gongdong Liao4State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaSentiment analysis is pivotal in advancing human–computer interaction (HCI) systems as it enables emotionally intelligent responses. While existing models show potential for HCI applications, current conversational datasets exhibit critical limitations in real-world deployment, particularly in capturing domain-specific emotional dynamics and context-sensitive behavioral patterns—constraints that hinder semantic comprehension and adaptive capabilities in task-driven HCI scenarios. To address these gaps, we present Multi-HM, the first multimodal emotion recognition dataset explicitly designed for human–machine consultation systems. It contains 2000 professionally annotated dialogues across 10 major HCI domains. Our dataset employs a five-dimensional annotation framework that systematically integrates textual, vocal, and visual modalities while simulating authentic HCI workflows to encode pragmatic behavioral cues and mission-critical emotional trajectories. Experiments demonstrate that Multi-HM-trained models achieve state-of-the-art performance in recognizing task-oriented affective states. This resource establishes a crucial foundation for developing human-centric AI systems that dynamically adapt to users’ evolving emotional needs.https://www.mdpi.com/2076-3417/15/8/4509emotion recognition in conversationmultimodal emotion recognitionmultimodal HCI datasethuman–machine consultation |
| spellingShingle | Yao Fu Qiong Liu Qing Song Pengzhou Zhang Gongdong Liao Multi-HM: A Chinese Multimodal Dataset and Fusion Framework for Emotion Recognition in Human–Machine Dialogue Systems Applied Sciences emotion recognition in conversation multimodal emotion recognition multimodal HCI dataset human–machine consultation |
| title | Multi-HM: A Chinese Multimodal Dataset and Fusion Framework for Emotion Recognition in Human–Machine Dialogue Systems |
| title_full | Multi-HM: A Chinese Multimodal Dataset and Fusion Framework for Emotion Recognition in Human–Machine Dialogue Systems |
| title_fullStr | Multi-HM: A Chinese Multimodal Dataset and Fusion Framework for Emotion Recognition in Human–Machine Dialogue Systems |
| title_full_unstemmed | Multi-HM: A Chinese Multimodal Dataset and Fusion Framework for Emotion Recognition in Human–Machine Dialogue Systems |
| title_short | Multi-HM: A Chinese Multimodal Dataset and Fusion Framework for Emotion Recognition in Human–Machine Dialogue Systems |
| title_sort | multi hm a chinese multimodal dataset and fusion framework for emotion recognition in human machine dialogue systems |
| topic | emotion recognition in conversation multimodal emotion recognition multimodal HCI dataset human–machine consultation |
| url | https://www.mdpi.com/2076-3417/15/8/4509 |
| work_keys_str_mv | AT yaofu multihmachinesemultimodaldatasetandfusionframeworkforemotionrecognitioninhumanmachinedialoguesystems AT qiongliu multihmachinesemultimodaldatasetandfusionframeworkforemotionrecognitioninhumanmachinedialoguesystems AT qingsong multihmachinesemultimodaldatasetandfusionframeworkforemotionrecognitioninhumanmachinedialoguesystems AT pengzhouzhang multihmachinesemultimodaldatasetandfusionframeworkforemotionrecognitioninhumanmachinedialoguesystems AT gongdongliao multihmachinesemultimodaldatasetandfusionframeworkforemotionrecognitioninhumanmachinedialoguesystems |