MS-RRBR: A Multi-Model Synergetic Framework for Restricted and Repetitive Behavior Recognition in Children with Autism
Restricted and Repetitive Behaviors (RRBs) are hallmark features of children with autism spectrum disorder (ASD) and are also one of the diagnostic criteria for the condition. Traditional methods of RRBs assessment through manual observation are limited by low diagnostic efficiency and uncertainty i...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-02-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/3/1577 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850068543979126784 |
|---|---|
| author | Yonggu Wang Yifan Shao Zengyi Yu Zihan Wang |
| author_facet | Yonggu Wang Yifan Shao Zengyi Yu Zihan Wang |
| author_sort | Yonggu Wang |
| collection | DOAJ |
| description | Restricted and Repetitive Behaviors (RRBs) are hallmark features of children with autism spectrum disorder (ASD) and are also one of the diagnostic criteria for the condition. Traditional methods of RRBs assessment through manual observation are limited by low diagnostic efficiency and uncertainty in outcomes. As a result, AI-assisted screening for autism has emerged as a promising research direction. In this study, we explore the synergy of visual foundation models and multimodal large language models (MLLMs), proposing a Multi-Model Synergistic Restricted and Repetitive Behavior Recognition method (MS-RRBR). Based on this method, we developed an interpretable multi-model autonomous question-answering system. To evaluate the effectiveness of our approach, we collected and annotated the Autism Restricted and Repetitive Behavior Dataset (ARRBD), which includes 10 ASD-related behaviors easily observable from various visual perspectives. Experimental results on the ARRBD dataset demonstrate that our multi-model collaboration outperforms single-model approaches, achieving the highest recognition accuracy of 94.94%. The MS-RRBR leverages the extensive linguistic knowledge of GPT-4o to enhance the zero-shot visual recognition capabilities of the MLLM, while also providing clear explanations for system decisions. This approach holds promise for providing timely, reliable, and accurate technical support for clinical diagnosis and educational rehabilitation in ASD. |
| format | Article |
| id | doaj-art-84b3122d56b247e68ef25cfb298c49e2 |
| institution | DOAJ |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-84b3122d56b247e68ef25cfb298c49e22025-08-20T02:48:02ZengMDPI AGApplied Sciences2076-34172025-02-01153157710.3390/app15031577MS-RRBR: A Multi-Model Synergetic Framework for Restricted and Repetitive Behavior Recognition in Children with AutismYonggu Wang0Yifan Shao1Zengyi Yu2Zihan Wang3College of Education, Zhejiang University of Technology, Hangzhou 310023, ChinaCollege of Education, Zhejiang University of Technology, Hangzhou 310023, ChinaCollege of Education, Zhejiang University of Technology, Hangzhou 310023, ChinaCollege of Education, Zhejiang University of Technology, Hangzhou 310023, ChinaRestricted and Repetitive Behaviors (RRBs) are hallmark features of children with autism spectrum disorder (ASD) and are also one of the diagnostic criteria for the condition. Traditional methods of RRBs assessment through manual observation are limited by low diagnostic efficiency and uncertainty in outcomes. As a result, AI-assisted screening for autism has emerged as a promising research direction. In this study, we explore the synergy of visual foundation models and multimodal large language models (MLLMs), proposing a Multi-Model Synergistic Restricted and Repetitive Behavior Recognition method (MS-RRBR). Based on this method, we developed an interpretable multi-model autonomous question-answering system. To evaluate the effectiveness of our approach, we collected and annotated the Autism Restricted and Repetitive Behavior Dataset (ARRBD), which includes 10 ASD-related behaviors easily observable from various visual perspectives. Experimental results on the ARRBD dataset demonstrate that our multi-model collaboration outperforms single-model approaches, achieving the highest recognition accuracy of 94.94%. The MS-RRBR leverages the extensive linguistic knowledge of GPT-4o to enhance the zero-shot visual recognition capabilities of the MLLM, while also providing clear explanations for system decisions. This approach holds promise for providing timely, reliable, and accurate technical support for clinical diagnosis and educational rehabilitation in ASD.https://www.mdpi.com/2076-3417/15/3/1577autism spectrum disorderrestricted and repetitive behaviorsaction recognitionmultimodal large language modelsGPT-4o |
| spellingShingle | Yonggu Wang Yifan Shao Zengyi Yu Zihan Wang MS-RRBR: A Multi-Model Synergetic Framework for Restricted and Repetitive Behavior Recognition in Children with Autism Applied Sciences autism spectrum disorder restricted and repetitive behaviors action recognition multimodal large language models GPT-4o |
| title | MS-RRBR: A Multi-Model Synergetic Framework for Restricted and Repetitive Behavior Recognition in Children with Autism |
| title_full | MS-RRBR: A Multi-Model Synergetic Framework for Restricted and Repetitive Behavior Recognition in Children with Autism |
| title_fullStr | MS-RRBR: A Multi-Model Synergetic Framework for Restricted and Repetitive Behavior Recognition in Children with Autism |
| title_full_unstemmed | MS-RRBR: A Multi-Model Synergetic Framework for Restricted and Repetitive Behavior Recognition in Children with Autism |
| title_short | MS-RRBR: A Multi-Model Synergetic Framework for Restricted and Repetitive Behavior Recognition in Children with Autism |
| title_sort | ms rrbr a multi model synergetic framework for restricted and repetitive behavior recognition in children with autism |
| topic | autism spectrum disorder restricted and repetitive behaviors action recognition multimodal large language models GPT-4o |
| url | https://www.mdpi.com/2076-3417/15/3/1577 |
| work_keys_str_mv | AT yongguwang msrrbramultimodelsynergeticframeworkforrestrictedandrepetitivebehaviorrecognitioninchildrenwithautism AT yifanshao msrrbramultimodelsynergeticframeworkforrestrictedandrepetitivebehaviorrecognitioninchildrenwithautism AT zengyiyu msrrbramultimodelsynergeticframeworkforrestrictedandrepetitivebehaviorrecognitioninchildrenwithautism AT zihanwang msrrbramultimodelsynergeticframeworkforrestrictedandrepetitivebehaviorrecognitioninchildrenwithautism |