Multitasking vision language models for vehicle plate recognition with VehiclePaliGemma
Abstract License Plate Recognition (LPR) automates vehicle identification using cameras and computer vision. It compares captured plates against databases to detect stolen vehicles, uninsured drivers, and crime suspects. Traditionally reliant on Optical Character Recognition (OCR), LPR faces challen...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-07-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-025-10774-9 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849763580505751552 |
|---|---|
| author | Nouar AlDahoul Myles Joshua Toledo Tan Raghava Reddy Tera Hezerul Abdul Karim Chee How Lim Manish Kumar Mishra Yasir Zaki |
| author_facet | Nouar AlDahoul Myles Joshua Toledo Tan Raghava Reddy Tera Hezerul Abdul Karim Chee How Lim Manish Kumar Mishra Yasir Zaki |
| author_sort | Nouar AlDahoul |
| collection | DOAJ |
| description | Abstract License Plate Recognition (LPR) automates vehicle identification using cameras and computer vision. It compares captured plates against databases to detect stolen vehicles, uninsured drivers, and crime suspects. Traditionally reliant on Optical Character Recognition (OCR), LPR faces challenges like noise, blurring, weather effects, and closely spaced characters, complicating accurate recognition. Existing LPR methods still require significant improvement, especially for distorted images. To fill this gap, we propose utilizing visual language models (VLMs) such as OpenAI GPT-4o (Generative Pre-trained Transformer 4 Omni), Google Gemini 1.5, Google PaliGemma (Pathways Language and Image model + Gemma model), Meta Llama (Large Language Model Meta AI) 3.2, Anthropic Claude 3.5 Sonnet, LLaVA (Large Language and Vision Assistant), NVIDIA VILA (Visual Language), and moondream2 to recognize such unclear plates with close characters. This paper evaluates the VLM’s capability to address the aforementioned problems. Additionally, we introduce “VehiclePaliGemma”, a fine-tuned Open-sourced PaliGemma VLM designed to recognize plates under challenging conditions. We compared our proposed VehiclePaliGemma with state-of-the-art methods and other VLMs using a dataset of Malaysian license plates collected under complex conditions. The results indicate that VehiclePaliGemma achieved superior performance with an accuracy of 87.6%. Moreover, it is able to predict the car’s plate at a speed of 7 frames per second using A100-80GB GPU. Finally, we explored the multitasking capability of VehiclePaliGemma model to accurately identify plates containing multiple cars of various models and colors, with plates positioned and oriented in different directions. |
| format | Article |
| id | doaj-art-780210b64cbb49e3bc6b104fca3640bd |
| institution | DOAJ |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-780210b64cbb49e3bc6b104fca3640bd2025-08-20T03:05:22ZengNature PortfolioScientific Reports2045-23222025-07-0115111510.1038/s41598-025-10774-9Multitasking vision language models for vehicle plate recognition with VehiclePaliGemmaNouar AlDahoul0Myles Joshua Toledo Tan1Raghava Reddy Tera2Hezerul Abdul Karim3Chee How Lim4Manish Kumar Mishra5Yasir Zaki6Computer Science, New York University Abu DhabiDepartment of Electrical and Computer Engineering, Herbert Wertheim College of Engineering, University of FloridaYo-Vivo CorporationCentre for Image and Vision Computing, Centre of Excellence for Artificial Intelligence, Faculty of Artificial Intelligence and Engineering, Multimedia UniversityTapway Sdn BhdTapway Sdn BhdComputer Science, New York University Abu DhabiAbstract License Plate Recognition (LPR) automates vehicle identification using cameras and computer vision. It compares captured plates against databases to detect stolen vehicles, uninsured drivers, and crime suspects. Traditionally reliant on Optical Character Recognition (OCR), LPR faces challenges like noise, blurring, weather effects, and closely spaced characters, complicating accurate recognition. Existing LPR methods still require significant improvement, especially for distorted images. To fill this gap, we propose utilizing visual language models (VLMs) such as OpenAI GPT-4o (Generative Pre-trained Transformer 4 Omni), Google Gemini 1.5, Google PaliGemma (Pathways Language and Image model + Gemma model), Meta Llama (Large Language Model Meta AI) 3.2, Anthropic Claude 3.5 Sonnet, LLaVA (Large Language and Vision Assistant), NVIDIA VILA (Visual Language), and moondream2 to recognize such unclear plates with close characters. This paper evaluates the VLM’s capability to address the aforementioned problems. Additionally, we introduce “VehiclePaliGemma”, a fine-tuned Open-sourced PaliGemma VLM designed to recognize plates under challenging conditions. We compared our proposed VehiclePaliGemma with state-of-the-art methods and other VLMs using a dataset of Malaysian license plates collected under complex conditions. The results indicate that VehiclePaliGemma achieved superior performance with an accuracy of 87.6%. Moreover, it is able to predict the car’s plate at a speed of 7 frames per second using A100-80GB GPU. Finally, we explored the multitasking capability of VehiclePaliGemma model to accurately identify plates containing multiple cars of various models and colors, with plates positioned and oriented in different directions.https://doi.org/10.1038/s41598-025-10774-9 |
| spellingShingle | Nouar AlDahoul Myles Joshua Toledo Tan Raghava Reddy Tera Hezerul Abdul Karim Chee How Lim Manish Kumar Mishra Yasir Zaki Multitasking vision language models for vehicle plate recognition with VehiclePaliGemma Scientific Reports |
| title | Multitasking vision language models for vehicle plate recognition with VehiclePaliGemma |
| title_full | Multitasking vision language models for vehicle plate recognition with VehiclePaliGemma |
| title_fullStr | Multitasking vision language models for vehicle plate recognition with VehiclePaliGemma |
| title_full_unstemmed | Multitasking vision language models for vehicle plate recognition with VehiclePaliGemma |
| title_short | Multitasking vision language models for vehicle plate recognition with VehiclePaliGemma |
| title_sort | multitasking vision language models for vehicle plate recognition with vehiclepaligemma |
| url | https://doi.org/10.1038/s41598-025-10774-9 |
| work_keys_str_mv | AT nouaraldahoul multitaskingvisionlanguagemodelsforvehicleplaterecognitionwithvehiclepaligemma AT mylesjoshuatoledotan multitaskingvisionlanguagemodelsforvehicleplaterecognitionwithvehiclepaligemma AT raghavareddytera multitaskingvisionlanguagemodelsforvehicleplaterecognitionwithvehiclepaligemma AT hezerulabdulkarim multitaskingvisionlanguagemodelsforvehicleplaterecognitionwithvehiclepaligemma AT cheehowlim multitaskingvisionlanguagemodelsforvehicleplaterecognitionwithvehiclepaligemma AT manishkumarmishra multitaskingvisionlanguagemodelsforvehicleplaterecognitionwithvehiclepaligemma AT yasirzaki multitaskingvisionlanguagemodelsforvehicleplaterecognitionwithvehiclepaligemma |