Leveraging multimodal large language model for multimodal sequential recommendation
Abstract Multimodal large language models (MLLMs) have demonstrated remarkable superiority in various vision-language tasks due to their unparalleled cross-modal comprehension capabilities and extensive world knowledge, offering promising research paradigms to address the insufficient information ex...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-08-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-14251-1 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849234666379280384 |
|---|---|
| author | Zhaoliang Wang Baisong Liu Weiming Huang Tingting Hao Huiqian Zhou Yuxin Guo |
| author_facet | Zhaoliang Wang Baisong Liu Weiming Huang Tingting Hao Huiqian Zhou Yuxin Guo |
| author_sort | Zhaoliang Wang |
| collection | DOAJ |
| description | Abstract Multimodal large language models (MLLMs) have demonstrated remarkable superiority in various vision-language tasks due to their unparalleled cross-modal comprehension capabilities and extensive world knowledge, offering promising research paradigms to address the insufficient information exploitation in conventional multimodal recommendation systems. Despite significant advances in existing recommendation approaches based on large language models, they still exhibit notable limitations in multimodal feature recognition and dynamic preference modeling, particularly in handling sequential data effectively and most of them predominantly rely on unimodal user-item interaction information, failing to adequately explore the cross-modal preference differences and the dynamic evolution of user interests within multimodal interaction sequences. These shortcomings have substantially prevented current research from fully unlocking the potential value of MLLMs within recommendation systems. To address these critical challenges, we present MLLM-SRec, a promising sequential recommendation architecture built upon MLLMs. Specifically, a novel multimodal feature fusion mechanism based on MLLMs is first established to generate unified semantic representations of items, which achieves semantic alignment between vision and text and effectively eliminates cross-modal differences and visual noise. Secondly, the temporal-aware user behavior comprehension module is designed to comprehensively capture the dynamic evolution law of user preference. Finally, by jointly modeling dynamic user preferences, user profiles, and multimodal information of target item, the supervised fine-tuning is combined with multistep Chain-of-Thought prompting optimization to facilitate effective knowledge transfer from the pre-trained multimodal model to the recommendation task, which effectively alleviates the problem of insufficient utilization of multimodal interaction data in generative recommendation. Experimental results demonstrate that our method achieves significant improvements over state-of-the-art baselines on four benchmark datasets and substantially enhances the precision of the recommendation results while exhibiting superior robustness and adaptability in multimodal sequential recommendation scenarios. These findings provide new methodological insights for multimodal sequence recommendation research and validate the potential of MLLMs for sequential recommendation tasks. Our code and data are available at https://github.com/MLLM-SRec. |
| format | Article |
| id | doaj-art-0f31031ea0904085a01d2833909dc4e7 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-08-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-0f31031ea0904085a01d2833909dc4e72025-08-20T04:03:03ZengNature PortfolioScientific Reports2045-23222025-08-0115111510.1038/s41598-025-14251-1Leveraging multimodal large language model for multimodal sequential recommendationZhaoliang Wang0Baisong Liu1Weiming Huang2Tingting Hao3Huiqian Zhou4Yuxin Guo5Faculty of Information Science and Engineering, Ningbo UniversityFaculty of Information Science and Engineering, Ningbo UniversityFaculty of Information Science and Engineering, Ningbo UniversitySchool of Material Science and Chemical Engineering, Ningbo UniversityState Key Laboratory of Chemo/Biosensing and Chemometrics, College of Chemistry and Chemical Engineering, Hunan UniversityDepartment of Biological Sciences, Xi’an Jiaotong-Liverpool UniversityAbstract Multimodal large language models (MLLMs) have demonstrated remarkable superiority in various vision-language tasks due to their unparalleled cross-modal comprehension capabilities and extensive world knowledge, offering promising research paradigms to address the insufficient information exploitation in conventional multimodal recommendation systems. Despite significant advances in existing recommendation approaches based on large language models, they still exhibit notable limitations in multimodal feature recognition and dynamic preference modeling, particularly in handling sequential data effectively and most of them predominantly rely on unimodal user-item interaction information, failing to adequately explore the cross-modal preference differences and the dynamic evolution of user interests within multimodal interaction sequences. These shortcomings have substantially prevented current research from fully unlocking the potential value of MLLMs within recommendation systems. To address these critical challenges, we present MLLM-SRec, a promising sequential recommendation architecture built upon MLLMs. Specifically, a novel multimodal feature fusion mechanism based on MLLMs is first established to generate unified semantic representations of items, which achieves semantic alignment between vision and text and effectively eliminates cross-modal differences and visual noise. Secondly, the temporal-aware user behavior comprehension module is designed to comprehensively capture the dynamic evolution law of user preference. Finally, by jointly modeling dynamic user preferences, user profiles, and multimodal information of target item, the supervised fine-tuning is combined with multistep Chain-of-Thought prompting optimization to facilitate effective knowledge transfer from the pre-trained multimodal model to the recommendation task, which effectively alleviates the problem of insufficient utilization of multimodal interaction data in generative recommendation. Experimental results demonstrate that our method achieves significant improvements over state-of-the-art baselines on four benchmark datasets and substantially enhances the precision of the recommendation results while exhibiting superior robustness and adaptability in multimodal sequential recommendation scenarios. These findings provide new methodological insights for multimodal sequence recommendation research and validate the potential of MLLMs for sequential recommendation tasks. Our code and data are available at https://github.com/MLLM-SRec.https://doi.org/10.1038/s41598-025-14251-1Recommender systemsSequential recommendationMultimodal recommendationMultimodal large language model |
| spellingShingle | Zhaoliang Wang Baisong Liu Weiming Huang Tingting Hao Huiqian Zhou Yuxin Guo Leveraging multimodal large language model for multimodal sequential recommendation Scientific Reports Recommender systems Sequential recommendation Multimodal recommendation Multimodal large language model |
| title | Leveraging multimodal large language model for multimodal sequential recommendation |
| title_full | Leveraging multimodal large language model for multimodal sequential recommendation |
| title_fullStr | Leveraging multimodal large language model for multimodal sequential recommendation |
| title_full_unstemmed | Leveraging multimodal large language model for multimodal sequential recommendation |
| title_short | Leveraging multimodal large language model for multimodal sequential recommendation |
| title_sort | leveraging multimodal large language model for multimodal sequential recommendation |
| topic | Recommender systems Sequential recommendation Multimodal recommendation Multimodal large language model |
| url | https://doi.org/10.1038/s41598-025-14251-1 |
| work_keys_str_mv | AT zhaoliangwang leveragingmultimodallargelanguagemodelformultimodalsequentialrecommendation AT baisongliu leveragingmultimodallargelanguagemodelformultimodalsequentialrecommendation AT weiminghuang leveragingmultimodallargelanguagemodelformultimodalsequentialrecommendation AT tingtinghao leveragingmultimodallargelanguagemodelformultimodalsequentialrecommendation AT huiqianzhou leveragingmultimodallargelanguagemodelformultimodalsequentialrecommendation AT yuxinguo leveragingmultimodallargelanguagemodelformultimodalsequentialrecommendation |