A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans
In recent years, advancements in the interaction and collaboration between humans and have garnered significant attention. Social intelligence plays a crucial role in facilitating natural interactions and seamless communication between humans and Artificial Intelligence (AI). To assess AI’s ability...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/25/2/477 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832587494488539136 |
|---|---|
| author | Erika Mori Yue Qiu Hirokatsu Kataoka Yoshimitsu Aoki |
| author_facet | Erika Mori Yue Qiu Hirokatsu Kataoka Yoshimitsu Aoki |
| author_sort | Erika Mori |
| collection | DOAJ |
| description | In recent years, advancements in the interaction and collaboration between humans and have garnered significant attention. Social intelligence plays a crucial role in facilitating natural interactions and seamless communication between humans and Artificial Intelligence (AI). To assess AI’s ability to understand human interactions and the components necessary for such comprehension, datasets like Social-IQ have been developed. However, these datasets often rely on a simplistic question-and-answer format and lack justifications for the provided answers. Furthermore, existing methods typically produce direct answers by selecting from predefined choices without generating intermediate outputs, which hampers interpretability and reliability. To address these limitations, we conducted a comprehensive evaluation of AI methods on a video-based Question Answering (QA) benchmark focused on human interactions, leveraging additional annotations related to human responses. Our analysis highlights significant differences between human and AI response patterns and underscores critical shortcomings in current benchmarks. We anticipate that these findings will guide the creation of more advanced datasets and represent an important step toward achieving natural communication between humans and AI. |
| format | Article |
| id | doaj-art-531453e2866d482ba39bf7b46725e921 |
| institution | Kabale University |
| issn | 1424-8220 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-531453e2866d482ba39bf7b46725e9212025-01-24T13:49:04ZengMDPI AGSensors1424-82202025-01-0125247710.3390/s25020477A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and HumansErika Mori0Yue Qiu1Hirokatsu Kataoka2Yoshimitsu Aoki3National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8560, JapanNational Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8560, JapanNational Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8560, JapanDepartment of Electronics and Electrical Engineering, Faculty of Science and Technology, Keio University, 3-14-1, Hiyoshi, Kohoku-ku, Yokohama 223-8522, JapanIn recent years, advancements in the interaction and collaboration between humans and have garnered significant attention. Social intelligence plays a crucial role in facilitating natural interactions and seamless communication between humans and Artificial Intelligence (AI). To assess AI’s ability to understand human interactions and the components necessary for such comprehension, datasets like Social-IQ have been developed. However, these datasets often rely on a simplistic question-and-answer format and lack justifications for the provided answers. Furthermore, existing methods typically produce direct answers by selecting from predefined choices without generating intermediate outputs, which hampers interpretability and reliability. To address these limitations, we conducted a comprehensive evaluation of AI methods on a video-based Question Answering (QA) benchmark focused on human interactions, leveraging additional annotations related to human responses. Our analysis highlights significant differences between human and AI response patterns and underscores critical shortcomings in current benchmarks. We anticipate that these findings will guide the creation of more advanced datasets and represent an important step toward achieving natural communication between humans and AI.https://www.mdpi.com/1424-8220/25/2/477human–robot interactionsocial intelligenceunderstanding human behavioremotion recognitionlarge language models (LLMs)VideoQA |
| spellingShingle | Erika Mori Yue Qiu Hirokatsu Kataoka Yoshimitsu Aoki A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans Sensors human–robot interaction social intelligence understanding human behavior emotion recognition large language models (LLMs) VideoQA |
| title | A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans |
| title_full | A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans |
| title_fullStr | A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans |
| title_full_unstemmed | A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans |
| title_short | A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans |
| title_sort | comprehensive analysis of a social intelligence dataset and response tendencies between large language models llms and humans |
| topic | human–robot interaction social intelligence understanding human behavior emotion recognition large language models (LLMs) VideoQA |
| url | https://www.mdpi.com/1424-8220/25/2/477 |
| work_keys_str_mv | AT erikamori acomprehensiveanalysisofasocialintelligencedatasetandresponsetendenciesbetweenlargelanguagemodelsllmsandhumans AT yueqiu acomprehensiveanalysisofasocialintelligencedatasetandresponsetendenciesbetweenlargelanguagemodelsllmsandhumans AT hirokatsukataoka acomprehensiveanalysisofasocialintelligencedatasetandresponsetendenciesbetweenlargelanguagemodelsllmsandhumans AT yoshimitsuaoki acomprehensiveanalysisofasocialintelligencedatasetandresponsetendenciesbetweenlargelanguagemodelsllmsandhumans AT erikamori comprehensiveanalysisofasocialintelligencedatasetandresponsetendenciesbetweenlargelanguagemodelsllmsandhumans AT yueqiu comprehensiveanalysisofasocialintelligencedatasetandresponsetendenciesbetweenlargelanguagemodelsllmsandhumans AT hirokatsukataoka comprehensiveanalysisofasocialintelligencedatasetandresponsetendenciesbetweenlargelanguagemodelsllmsandhumans AT yoshimitsuaoki comprehensiveanalysisofasocialintelligencedatasetandresponsetendenciesbetweenlargelanguagemodelsllmsandhumans |