Applications of Large Language Models and Multimodal Large Models in Autonomous Driving: A Comprehensive Review
The rapid development of large language models (LLMs) and multimodal large models (MLMs) has introduced transformative opportunities for autonomous driving systems. These advanced models provide robust support for the realization of more intelligent, safer, and efficient autonomous driving. In this...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-03-01
|
| Series: | Drones |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2504-446X/9/4/238 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850183296863961088 |
|---|---|
| author | Jing Li Jingyuan Li Guo Yang Lie Yang Haozhuang Chi Lichao Yang |
| author_facet | Jing Li Jingyuan Li Guo Yang Lie Yang Haozhuang Chi Lichao Yang |
| author_sort | Jing Li |
| collection | DOAJ |
| description | The rapid development of large language models (LLMs) and multimodal large models (MLMs) has introduced transformative opportunities for autonomous driving systems. These advanced models provide robust support for the realization of more intelligent, safer, and efficient autonomous driving. In this paper, we present a systematic review on the integration of LLMs and MLMs in autonomous driving systems. First, we provide an overview of the evolution of LLMs and MLMs, along with a detailed analysis of the architecture of autonomous driving systems. Next, we explore the applications of LLMs and MLMs in key components such as perception, prediction, decision making, planning, multitask processing, and human–machine interaction. Additionally, this paper reviews the core technologies involved in integrating LLMs and MLMs with autonomous driving systems, including multimodal fusion, knowledge distillation, prompt engineering, and supervised fine tuning. Finally, we provide an in-depth analysis of the major challenges faced by autonomous driving systems powered by large models, offering new perspectives for future research. Compared to existing review articles, this paper not only systematically examines the specific applications of LLMs and MLMs in autonomous driving systems but also delves into the key technologies and potential challenges involved in their integration. By comprehensively organizing and analyzing the current literature, this review highlights the application potential of large models in autonomous driving and offers insights and recommendations for improving system safety and efficiency. |
| format | Article |
| id | doaj-art-53cb6affb991461ca0dffe0b298f4c0f |
| institution | OA Journals |
| issn | 2504-446X |
| language | English |
| publishDate | 2025-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Drones |
| spelling | doaj-art-53cb6affb991461ca0dffe0b298f4c0f2025-08-20T02:17:25ZengMDPI AGDrones2504-446X2025-03-019423810.3390/drones9040238Applications of Large Language Models and Multimodal Large Models in Autonomous Driving: A Comprehensive ReviewJing Li0Jingyuan Li1Guo Yang2Lie Yang3Haozhuang Chi4Lichao Yang5Independent Researcher, Nanchang 330000, ChinaSchool of Vehicle and Mobility, Tsinghua University, Beijing 100084, ChinaDepartment of Industrial and Manufacturing Systems Engineering, The University of Hong Kong, Hong Kong 999077, ChinaSchool of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798, SingaporeSchool of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798, SingaporeFaculty of Engineering and Applied Science, Cranfield University, Cranfield MK43 0AL, UKThe rapid development of large language models (LLMs) and multimodal large models (MLMs) has introduced transformative opportunities for autonomous driving systems. These advanced models provide robust support for the realization of more intelligent, safer, and efficient autonomous driving. In this paper, we present a systematic review on the integration of LLMs and MLMs in autonomous driving systems. First, we provide an overview of the evolution of LLMs and MLMs, along with a detailed analysis of the architecture of autonomous driving systems. Next, we explore the applications of LLMs and MLMs in key components such as perception, prediction, decision making, planning, multitask processing, and human–machine interaction. Additionally, this paper reviews the core technologies involved in integrating LLMs and MLMs with autonomous driving systems, including multimodal fusion, knowledge distillation, prompt engineering, and supervised fine tuning. Finally, we provide an in-depth analysis of the major challenges faced by autonomous driving systems powered by large models, offering new perspectives for future research. Compared to existing review articles, this paper not only systematically examines the specific applications of LLMs and MLMs in autonomous driving systems but also delves into the key technologies and potential challenges involved in their integration. By comprehensively organizing and analyzing the current literature, this review highlights the application potential of large models in autonomous driving and offers insights and recommendations for improving system safety and efficiency.https://www.mdpi.com/2504-446X/9/4/238large language modelsmultimodal large modelsautonomous drivingcomprehensive review |
| spellingShingle | Jing Li Jingyuan Li Guo Yang Lie Yang Haozhuang Chi Lichao Yang Applications of Large Language Models and Multimodal Large Models in Autonomous Driving: A Comprehensive Review Drones large language models multimodal large models autonomous driving comprehensive review |
| title | Applications of Large Language Models and Multimodal Large Models in Autonomous Driving: A Comprehensive Review |
| title_full | Applications of Large Language Models and Multimodal Large Models in Autonomous Driving: A Comprehensive Review |
| title_fullStr | Applications of Large Language Models and Multimodal Large Models in Autonomous Driving: A Comprehensive Review |
| title_full_unstemmed | Applications of Large Language Models and Multimodal Large Models in Autonomous Driving: A Comprehensive Review |
| title_short | Applications of Large Language Models and Multimodal Large Models in Autonomous Driving: A Comprehensive Review |
| title_sort | applications of large language models and multimodal large models in autonomous driving a comprehensive review |
| topic | large language models multimodal large models autonomous driving comprehensive review |
| url | https://www.mdpi.com/2504-446X/9/4/238 |
| work_keys_str_mv | AT jingli applicationsoflargelanguagemodelsandmultimodallargemodelsinautonomousdrivingacomprehensivereview AT jingyuanli applicationsoflargelanguagemodelsandmultimodallargemodelsinautonomousdrivingacomprehensivereview AT guoyang applicationsoflargelanguagemodelsandmultimodallargemodelsinautonomousdrivingacomprehensivereview AT lieyang applicationsoflargelanguagemodelsandmultimodallargemodelsinautonomousdrivingacomprehensivereview AT haozhuangchi applicationsoflargelanguagemodelsandmultimodallargemodelsinautonomousdrivingacomprehensivereview AT lichaoyang applicationsoflargelanguagemodelsandmultimodallargemodelsinautonomousdrivingacomprehensivereview |