Explainable human-centered traits from head motion and facial expression dynamics.
We explore the efficacy of multimodal behavioral cues for explainable prediction of personality and interview-specific traits. We utilize elementary head-motion units named kinemes, atomic facial movements termed action units and speech features to estimate these human-centered traits. Empirical res...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2025-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0313883 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850284795762835456 |
|---|---|
| author | Surbhi Madan Monika Gahalawat Tanaya Guha Roland Goecke Ramanathan Subramanian |
| author_facet | Surbhi Madan Monika Gahalawat Tanaya Guha Roland Goecke Ramanathan Subramanian |
| author_sort | Surbhi Madan |
| collection | DOAJ |
| description | We explore the efficacy of multimodal behavioral cues for explainable prediction of personality and interview-specific traits. We utilize elementary head-motion units named kinemes, atomic facial movements termed action units and speech features to estimate these human-centered traits. Empirical results confirm that kinemes and action units enable discovery of multiple trait-specific behaviors while also enabling explainability in support of the predictions. For fusing cues, we explore decision and feature-level fusion, and an additive attention-based fusion strategy which quantifies the relative importance of the three modalities for trait prediction. Examining various long-short term memory (LSTM) architectures for classification and regression on the MIT Interview and First Impressions Candidate Screening (FICS) datasets, we note that: (1) Multimodal approaches outperform unimodal counterparts, achieving the highest PCC of 0.98 for Excited-Friendly traits in MIT and 0.57 for Extraversion in FICS; (2) Efficient trait predictions and plausible explanations are achieved with both unimodal and multimodal approaches, and (3) Following the thin-slice approach, effective trait prediction is achieved even from two-second behavioral snippets. Our implementation code is available at: https://github.com/deepsurbhi8/Explainable_Human_Traits_Prediction. |
| format | Article |
| id | doaj-art-18e30da6868645e8bf6fa21f5d4fe672 |
| institution | OA Journals |
| issn | 1932-6203 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-18e30da6868645e8bf6fa21f5d4fe6722025-08-20T01:47:28ZengPublic Library of Science (PLoS)PLoS ONE1932-62032025-01-01201e031388310.1371/journal.pone.0313883Explainable human-centered traits from head motion and facial expression dynamics.Surbhi MadanMonika GahalawatTanaya GuhaRoland GoeckeRamanathan SubramanianWe explore the efficacy of multimodal behavioral cues for explainable prediction of personality and interview-specific traits. We utilize elementary head-motion units named kinemes, atomic facial movements termed action units and speech features to estimate these human-centered traits. Empirical results confirm that kinemes and action units enable discovery of multiple trait-specific behaviors while also enabling explainability in support of the predictions. For fusing cues, we explore decision and feature-level fusion, and an additive attention-based fusion strategy which quantifies the relative importance of the three modalities for trait prediction. Examining various long-short term memory (LSTM) architectures for classification and regression on the MIT Interview and First Impressions Candidate Screening (FICS) datasets, we note that: (1) Multimodal approaches outperform unimodal counterparts, achieving the highest PCC of 0.98 for Excited-Friendly traits in MIT and 0.57 for Extraversion in FICS; (2) Efficient trait predictions and plausible explanations are achieved with both unimodal and multimodal approaches, and (3) Following the thin-slice approach, effective trait prediction is achieved even from two-second behavioral snippets. Our implementation code is available at: https://github.com/deepsurbhi8/Explainable_Human_Traits_Prediction.https://doi.org/10.1371/journal.pone.0313883 |
| spellingShingle | Surbhi Madan Monika Gahalawat Tanaya Guha Roland Goecke Ramanathan Subramanian Explainable human-centered traits from head motion and facial expression dynamics. PLoS ONE |
| title | Explainable human-centered traits from head motion and facial expression dynamics. |
| title_full | Explainable human-centered traits from head motion and facial expression dynamics. |
| title_fullStr | Explainable human-centered traits from head motion and facial expression dynamics. |
| title_full_unstemmed | Explainable human-centered traits from head motion and facial expression dynamics. |
| title_short | Explainable human-centered traits from head motion and facial expression dynamics. |
| title_sort | explainable human centered traits from head motion and facial expression dynamics |
| url | https://doi.org/10.1371/journal.pone.0313883 |
| work_keys_str_mv | AT surbhimadan explainablehumancenteredtraitsfromheadmotionandfacialexpressiondynamics AT monikagahalawat explainablehumancenteredtraitsfromheadmotionandfacialexpressiondynamics AT tanayaguha explainablehumancenteredtraitsfromheadmotionandfacialexpressiondynamics AT rolandgoecke explainablehumancenteredtraitsfromheadmotionandfacialexpressiondynamics AT ramanathansubramanian explainablehumancenteredtraitsfromheadmotionandfacialexpressiondynamics |