BrAIn Jam: neural signal-informed adaptive system for drumming collaboration with an AI-driven virtual musician
Collaboration between improvising musicians requires a dynamic exchange of subtleties in human musical communication. Many musicians can intuit this information, however, translating this knowledge to embodied computer-driven musicianship systems—be they robotic or virtual musicians—remains an ongoi...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-08-01
|
| Series: | Frontiers in Computer Science |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fcomp.2025.1570249/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850026790703071232 |
|---|---|
| author | Torin Hopkins Torin Hopkins Ruojia Sun Suibi Che Chuan Weng Shih-Yu Ma Shih-Yu Ma James Crum Leanne Hirshfield Ellen Yi-Luen Do Ellen Yi-Luen Do |
| author_facet | Torin Hopkins Torin Hopkins Ruojia Sun Suibi Che Chuan Weng Shih-Yu Ma Shih-Yu Ma James Crum Leanne Hirshfield Ellen Yi-Luen Do Ellen Yi-Luen Do |
| author_sort | Torin Hopkins |
| collection | DOAJ |
| description | Collaboration between improvising musicians requires a dynamic exchange of subtleties in human musical communication. Many musicians can intuit this information, however, translating this knowledge to embodied computer-driven musicianship systems—be they robotic or virtual musicians—remains an ongoing challenge. Methods of communicating musical information to computer-driven musicianship systems have traditionally been accomplished using an array of sensing techniques such as MIDI, audio, and video. However, utilizing musical information from the human brain has only been explored in limited social and musical contexts. This paper presents “BrAIn Jam,” utilizing functional near-infrared spectroscopy to monitor human drummers' brain states during musical collaboration with an AI-driven virtual musician. Our system includes a real-time algorithm for preprocessing and classifying brain data, enabling dynamic AI rhythm adjustments based on neural signal processing. Our formative study is conducted in two phases: (1) training individualized machine learning models using data collected during a controlled experiment, and (2) using these models to inform an embodied AI-driven virtual musician in a real-time improvised drumming collaboration. In this paper, we discuss our experimental approach to isolating a network of brain areas involved in music improvisation with embodied AI-driven musicians, a comparative analysis of several machine learning models, and post hoc analysis of brain activation to corroborate our findings. We then synthesize findings from interviews with our participants and report on the challenges and opportunities for designing music systems with functional near-infrared spectroscopy, as well as the applicability of other physiological sensing techniques for human and AI-driven musician communication. |
| format | Article |
| id | doaj-art-52aa91d3e2c54cd0bf9d07244279af98 |
| institution | DOAJ |
| issn | 2624-9898 |
| language | English |
| publishDate | 2025-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Computer Science |
| spelling | doaj-art-52aa91d3e2c54cd0bf9d07244279af982025-08-20T03:00:25ZengFrontiers Media S.A.Frontiers in Computer Science2624-98982025-08-01710.3389/fcomp.2025.15702491570249BrAIn Jam: neural signal-informed adaptive system for drumming collaboration with an AI-driven virtual musicianTorin Hopkins0Torin Hopkins1Ruojia Sun2Suibi Che Chuan Weng3Shih-Yu Ma4Shih-Yu Ma5James Crum6Leanne Hirshfield7Ellen Yi-Luen Do8Ellen Yi-Luen Do9ATLAS Institute, University of Colorado Boulder, Boulder, CO, United StatesInstitute of Cognitive Science, University of Colorado Boulder, Boulder, CO, United StatesATLAS Institute, University of Colorado Boulder, Boulder, CO, United StatesATLAS Institute, University of Colorado Boulder, Boulder, CO, United StatesATLAS Institute, University of Colorado Boulder, Boulder, CO, United StatesInstitute of Cognitive Science, University of Colorado Boulder, Boulder, CO, United StatesInstitute of Cognitive Science, University of Colorado Boulder, Boulder, CO, United StatesInstitute of Cognitive Science, University of Colorado Boulder, Boulder, CO, United StatesATLAS Institute, University of Colorado Boulder, Boulder, CO, United StatesInstitute of Cognitive Science, University of Colorado Boulder, Boulder, CO, United StatesCollaboration between improvising musicians requires a dynamic exchange of subtleties in human musical communication. Many musicians can intuit this information, however, translating this knowledge to embodied computer-driven musicianship systems—be they robotic or virtual musicians—remains an ongoing challenge. Methods of communicating musical information to computer-driven musicianship systems have traditionally been accomplished using an array of sensing techniques such as MIDI, audio, and video. However, utilizing musical information from the human brain has only been explored in limited social and musical contexts. This paper presents “BrAIn Jam,” utilizing functional near-infrared spectroscopy to monitor human drummers' brain states during musical collaboration with an AI-driven virtual musician. Our system includes a real-time algorithm for preprocessing and classifying brain data, enabling dynamic AI rhythm adjustments based on neural signal processing. Our formative study is conducted in two phases: (1) training individualized machine learning models using data collected during a controlled experiment, and (2) using these models to inform an embodied AI-driven virtual musician in a real-time improvised drumming collaboration. In this paper, we discuss our experimental approach to isolating a network of brain areas involved in music improvisation with embodied AI-driven musicians, a comparative analysis of several machine learning models, and post hoc analysis of brain activation to corroborate our findings. We then synthesize findings from interviews with our participants and report on the challenges and opportunities for designing music systems with functional near-infrared spectroscopy, as well as the applicability of other physiological sensing techniques for human and AI-driven musician communication.https://www.frontiersin.org/articles/10.3389/fcomp.2025.1570249/fullfunctional near-infrared spectroscopy (fNIRS)brain-computer interfacesembodied AImusicneurosciencemachine learning |
| spellingShingle | Torin Hopkins Torin Hopkins Ruojia Sun Suibi Che Chuan Weng Shih-Yu Ma Shih-Yu Ma James Crum Leanne Hirshfield Ellen Yi-Luen Do Ellen Yi-Luen Do BrAIn Jam: neural signal-informed adaptive system for drumming collaboration with an AI-driven virtual musician Frontiers in Computer Science functional near-infrared spectroscopy (fNIRS) brain-computer interfaces embodied AI music neuroscience machine learning |
| title | BrAIn Jam: neural signal-informed adaptive system for drumming collaboration with an AI-driven virtual musician |
| title_full | BrAIn Jam: neural signal-informed adaptive system for drumming collaboration with an AI-driven virtual musician |
| title_fullStr | BrAIn Jam: neural signal-informed adaptive system for drumming collaboration with an AI-driven virtual musician |
| title_full_unstemmed | BrAIn Jam: neural signal-informed adaptive system for drumming collaboration with an AI-driven virtual musician |
| title_short | BrAIn Jam: neural signal-informed adaptive system for drumming collaboration with an AI-driven virtual musician |
| title_sort | brain jam neural signal informed adaptive system for drumming collaboration with an ai driven virtual musician |
| topic | functional near-infrared spectroscopy (fNIRS) brain-computer interfaces embodied AI music neuroscience machine learning |
| url | https://www.frontiersin.org/articles/10.3389/fcomp.2025.1570249/full |
| work_keys_str_mv | AT torinhopkins brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT torinhopkins brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT ruojiasun brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT suibichechuanweng brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT shihyuma brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT shihyuma brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT jamescrum brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT leannehirshfield brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT ellenyiluendo brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician AT ellenyiluendo brainjamneuralsignalinformedadaptivesystemfordrummingcollaborationwithanaidrivenvirtualmusician |