Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience
Researchers are increasingly using machine learning to study physiological markers of emotion. We evaluated the promises and limitations of this approach via a big team science competition. Twelve teams competed to predict self-reported affective experiences using a multi-modal set of peripheral ner...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2025-06-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/10.1098/rsos.241778 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849425928804892672 |
|---|---|
| author | Nicholas A. Coles Bartosz Perz Maciej Behnke Johannes C. Eichstaedt Soo Hyung Kim Tu N. Vu Chirag Raman Julian Tejada Van-Thong Huynh Guangyi Zhang Tanming Cui Sharanyak Podder Rushi Chavda Shubham Pandey Arpit Upadhyay Jorge I. Padilla-Buritica Carlos J. Barrera Causil Linying Ji Felix Dollack Kiyoshi Kiyokawa Huakun Liu Monica Perusquia-Hernandez Hideaki Uchiyama Xin Wei Houwei Cao Ziqing Yang Alessia Iancarelli Kieran McVeigh Yiyu Wang Isabel M. Berwian Jamie C. Chiu Dan-Mircea Mirea Erik C. Nook Henna I. Vartiainen Claire Whiting Young Won Cho Sy-Miin Chow Zachary F. Fisher Yanling Li Xiaoyue Xiong Yuqi Shen Enzo Tagliazucchi Leandro A. Bugnon Raydonal Ospina Nicolas M. Bruno Tomas A. D'Amelio Federico Zamberlan Luis R. Mercado Diaz Javier O. Pinzon-Arenas Hugo F. Posada-Quintero Maneesh Bilalpur Saurabh Hinduja Fernando Marmolejo-Ramos Shaun Canavan Liza Jivnani Stanisław Saganowski |
| author_facet | Nicholas A. Coles Bartosz Perz Maciej Behnke Johannes C. Eichstaedt Soo Hyung Kim Tu N. Vu Chirag Raman Julian Tejada Van-Thong Huynh Guangyi Zhang Tanming Cui Sharanyak Podder Rushi Chavda Shubham Pandey Arpit Upadhyay Jorge I. Padilla-Buritica Carlos J. Barrera Causil Linying Ji Felix Dollack Kiyoshi Kiyokawa Huakun Liu Monica Perusquia-Hernandez Hideaki Uchiyama Xin Wei Houwei Cao Ziqing Yang Alessia Iancarelli Kieran McVeigh Yiyu Wang Isabel M. Berwian Jamie C. Chiu Dan-Mircea Mirea Erik C. Nook Henna I. Vartiainen Claire Whiting Young Won Cho Sy-Miin Chow Zachary F. Fisher Yanling Li Xiaoyue Xiong Yuqi Shen Enzo Tagliazucchi Leandro A. Bugnon Raydonal Ospina Nicolas M. Bruno Tomas A. D'Amelio Federico Zamberlan Luis R. Mercado Diaz Javier O. Pinzon-Arenas Hugo F. Posada-Quintero Maneesh Bilalpur Saurabh Hinduja Fernando Marmolejo-Ramos Shaun Canavan Liza Jivnani Stanisław Saganowski |
| author_sort | Nicholas A. Coles |
| collection | DOAJ |
| description | Researchers are increasingly using machine learning to study physiological markers of emotion. We evaluated the promises and limitations of this approach via a big team science competition. Twelve teams competed to predict self-reported affective experiences using a multi-modal set of peripheral nervous system measures. Models were trained and tested in multiple ways: with data divided by participants, targeted emotion, inductions, and time. In 100% of tests, teams outperformed baseline models that made random predictions. In 46% of tests, teams also outperformed baseline models that relied on the simple average of ratings from training datasets. More notably, results uncovered a methodological challenge: multiplicative constraints on generalizability. Inferences about the accuracy and theoretical implications of machine learning efforts depended not only on their architecture, but also how they were trained, tested, and evaluated. For example, some teams performed better when tested on observations from the same (vs. different) subjects seen during training. Such results could be interpreted as evidence against claims of universality. However, such conclusions would be premature because other teams exhibited the opposite pattern. Taken together, results illustrate how big team science can be leveraged to understand the promises and limitations of machine learning methods in affective science and beyond. |
| format | Article |
| id | doaj-art-b4426e0d364e46bfaece912508f4a8af |
| institution | Kabale University |
| issn | 2054-5703 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj-art-b4426e0d364e46bfaece912508f4a8af2025-08-20T03:29:35ZengThe Royal SocietyRoyal Society Open Science2054-57032025-06-0112610.1098/rsos.241778Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experienceNicholas A. Coles0Bartosz Perz1Maciej Behnke2Johannes C. Eichstaedt3Soo Hyung Kim4Tu N. Vu5Chirag Raman6Julian Tejada7Van-Thong Huynh8Guangyi Zhang9Tanming Cui10Sharanyak Podder11Rushi Chavda12Shubham Pandey13Arpit Upadhyay14Jorge I. Padilla-Buritica15Carlos J. Barrera Causil16Linying Ji17Felix Dollack18Kiyoshi Kiyokawa19Huakun Liu20Monica Perusquia-Hernandez21Hideaki Uchiyama22Xin Wei23Houwei Cao24Ziqing Yang25Alessia Iancarelli26Kieran McVeigh27Yiyu Wang28Isabel M. Berwian29Jamie C. Chiu30Dan-Mircea Mirea31Erik C. Nook32Henna I. Vartiainen33Claire Whiting34Young Won Cho35Sy-Miin Chow36Zachary F. Fisher37Yanling Li38Xiaoyue Xiong39Yuqi Shen40Enzo Tagliazucchi41Leandro A. Bugnon42Raydonal Ospina43Nicolas M. Bruno44Tomas A. D'Amelio45Federico Zamberlan46Luis R. Mercado Diaz47Javier O. Pinzon-Arenas48Hugo F. Posada-Quintero49Maneesh Bilalpur50Saurabh Hinduja51Fernando Marmolejo-Ramos52Shaun Canavan53Liza Jivnani54Stanisław Saganowski55University of Florida, Gainesville, FL, USAWrocław University of Science and Technology, Wroclaw, Województwo Dolnośląskie, PolandAdam Mickiewicz University, Poznan, PolandStanford University, Stanford, CA, USAChonnam National University, Gwangju, Jeollanam-do, Republic of KoreaChonnam National University, Gwangju, Jeollanam-do, Republic of KoreaDelft University of Technology, Delft, Zuid-Holland, The NetherlandsFederal University of Sergipe, Sao Cristovao, Sergipe, BrazilFPT University, Hanoi, VietnamHarvard Medical School, Boston, MA, USAIndependent Researcher, State College, PA, USAIndian Institute of Science Education and Research Bhopal, Bhopal, Madhya Pradesh, IndiaIndian Institute of Technology Bombay, Mumbai, Maharashtra, IndiaIndian Institute of Technology Bombay, Mumbai, Maharashtra, IndiaIndian Institute of Technology Bombay, Mumbai, Maharashtra, IndiaInstitución Universitaria ITM, Medellín, ColombiaInstitución Universitaria ITM, Medellín, ColombiaMontana State University, Bozeman, MT, USANara Institute of Science and Technology, Ikoma, Nara, JapanNara Institute of Science and Technology, Ikoma, Nara, JapanNara Institute of Science and Technology, Ikoma, Nara, JapanNara Institute of Science and Technology, Ikoma, Nara, JapanNara Institute of Science and Technology, Ikoma, Nara, JapanNara Institute of Science and Technology, Ikoma, Nara, JapanNew York Institute of Technology, Old Westbury, NY, USANew York Institute of Technology, Old Westbury, NY, USANortheastern University—Boston Campus, Boston, MA, USANortheastern University—Boston Campus, Boston, MA, USANortheastern University—Boston Campus, Boston, MA, USAPrinceton University, Princeton, NJ, USAPrinceton University, Princeton, NJ, USAPrinceton University, Princeton, NJ, USAPrinceton University, Princeton, NJ, USAPrinceton University, Princeton, NJ, USAPrinceton University, Princeton, NJ, USAThe Pennsylvania State University, University Park, PA, USAThe Pennsylvania State University, University Park, PA, USAThe Pennsylvania State University, University Park, PA, USAThe Pennsylvania State University, University Park, PA, USAThe Pennsylvania State University, University Park, PA, USAThe Pennsylvania State University, University Park, PA, USAUniversidad Adolfo Ibanez, Penalolen, ChileResearch Institute for Signals, Systems and Computational Intelligence, sinc(i), FICH-UNL, CONICET, Santa Fe, ArgentinaUniversidade Federal da Bahia, Salvador, BrazilUniversity of Buenos Aires, Buenos Aires, ArgentinaUniversity of Buenos Aires, Buenos Aires, ArgentinaUniversity of Buenos Aires, Buenos Aires, ArgentinaUniversity of Connecticut, Storrs, CT, USAUniversity of Connecticut, Storrs, CT, USAUniversity of Connecticut, Storrs, CT, USAUniversity of Pittsburgh, Pittsburgh, PA, USAUniversity of Akron, Akron, OH, USAFlinders University, Adelaide, South Australia, AustraliaUniversity of South Florida, Tampa, FL, USAUniversity of South Florida, Tampa, FL, USAWrocław University of Science and Technology, Wroclaw, Województwo Dolnośląskie, PolandResearchers are increasingly using machine learning to study physiological markers of emotion. We evaluated the promises and limitations of this approach via a big team science competition. Twelve teams competed to predict self-reported affective experiences using a multi-modal set of peripheral nervous system measures. Models were trained and tested in multiple ways: with data divided by participants, targeted emotion, inductions, and time. In 100% of tests, teams outperformed baseline models that made random predictions. In 46% of tests, teams also outperformed baseline models that relied on the simple average of ratings from training datasets. More notably, results uncovered a methodological challenge: multiplicative constraints on generalizability. Inferences about the accuracy and theoretical implications of machine learning efforts depended not only on their architecture, but also how they were trained, tested, and evaluated. For example, some teams performed better when tested on observations from the same (vs. different) subjects seen during training. Such results could be interpreted as evidence against claims of universality. However, such conclusions would be premature because other teams exhibited the opposite pattern. Taken together, results illustrate how big team science can be leveraged to understand the promises and limitations of machine learning methods in affective science and beyond.https://royalsocietypublishing.org/doi/10.1098/rsos.241778big team sciencemachine learningemotionphysiologygeneralizabilityaffective computing |
| spellingShingle | Nicholas A. Coles Bartosz Perz Maciej Behnke Johannes C. Eichstaedt Soo Hyung Kim Tu N. Vu Chirag Raman Julian Tejada Van-Thong Huynh Guangyi Zhang Tanming Cui Sharanyak Podder Rushi Chavda Shubham Pandey Arpit Upadhyay Jorge I. Padilla-Buritica Carlos J. Barrera Causil Linying Ji Felix Dollack Kiyoshi Kiyokawa Huakun Liu Monica Perusquia-Hernandez Hideaki Uchiyama Xin Wei Houwei Cao Ziqing Yang Alessia Iancarelli Kieran McVeigh Yiyu Wang Isabel M. Berwian Jamie C. Chiu Dan-Mircea Mirea Erik C. Nook Henna I. Vartiainen Claire Whiting Young Won Cho Sy-Miin Chow Zachary F. Fisher Yanling Li Xiaoyue Xiong Yuqi Shen Enzo Tagliazucchi Leandro A. Bugnon Raydonal Ospina Nicolas M. Bruno Tomas A. D'Amelio Federico Zamberlan Luis R. Mercado Diaz Javier O. Pinzon-Arenas Hugo F. Posada-Quintero Maneesh Bilalpur Saurabh Hinduja Fernando Marmolejo-Ramos Shaun Canavan Liza Jivnani Stanisław Saganowski Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience Royal Society Open Science big team science machine learning emotion physiology generalizability affective computing |
| title | Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience |
| title_full | Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience |
| title_fullStr | Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience |
| title_full_unstemmed | Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience |
| title_short | Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience |
| title_sort | big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience |
| topic | big team science machine learning emotion physiology generalizability affective computing |
| url | https://royalsocietypublishing.org/doi/10.1098/rsos.241778 |
| work_keys_str_mv | AT nicholasacoles bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT bartoszperz bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT maciejbehnke bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT johannesceichstaedt bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT soohyungkim bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT tunvu bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT chiragraman bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT juliantejada bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT vanthonghuynh bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT guangyizhang bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT tanmingcui bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT sharanyakpodder bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT rushichavda bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT shubhampandey bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT arpitupadhyay bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT jorgeipadillaburitica bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT carlosjbarreracausil bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT linyingji bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT felixdollack bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT kiyoshikiyokawa bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT huakunliu bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT monicaperusquiahernandez bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT hideakiuchiyama bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT xinwei bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT houweicao bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT ziqingyang bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT alessiaiancarelli bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT kieranmcveigh bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT yiyuwang bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT isabelmberwian bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT jamiecchiu bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT danmirceamirea bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT erikcnook bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT hennaivartiainen bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT clairewhiting bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT youngwoncho bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT symiinchow bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT zacharyffisher bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT yanlingli bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT xiaoyuexiong bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT yuqishen bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT enzotagliazucchi bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT leandroabugnon bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT raydonalospina bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT nicolasmbruno bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT tomasadamelio bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT federicozamberlan bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT luisrmercadodiaz bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT javieropinzonarenas bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT hugofposadaquintero bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT maneeshbilalpur bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT saurabhhinduja bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT fernandomarmolejoramos bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT shauncanavan bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT lizajivnani bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience AT stanisławsaganowski bigteamsciencerevealspromisesandlimitationsofmachinelearningeffortstomodelphysiologicalmarkersofaffectiveexperience |