Nighttime traffic object detection via adaptively integrating event and frame domains
Intelligent perception is crucial in Intelligent Transportation Systems (ITS), with vision cameras as critical components. However, traditional RGB cameras exhibit a significant decline in performance when capturing nighttime traffic scenes, limiting their effectiveness in supporting ITS. In contras...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
KeAi Communications Co. Ltd.
2025-07-01
|
| Series: | Fundamental Research |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2667325823002376 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850066478572765184 |
|---|---|
| author | Yu Jiang Yuehang Wang Minghao Zhao Yongji Zhang Hong Qi |
| author_facet | Yu Jiang Yuehang Wang Minghao Zhao Yongji Zhang Hong Qi |
| author_sort | Yu Jiang |
| collection | DOAJ |
| description | Intelligent perception is crucial in Intelligent Transportation Systems (ITS), with vision cameras as critical components. However, traditional RGB cameras exhibit a significant decline in performance when capturing nighttime traffic scenes, limiting their effectiveness in supporting ITS. In contrast, event cameras possess a high dynamic range (140 dB vs. 60 dB for traditional cameras), enabling them to overcome frame degradation in low-light conditions. Recently, multimodal learning paradigms have made substantial progress in various vision tasks, such as image-text retrieval. Motivated by this progress, we propose an adaptive selection and fusion detection method that leverages both event and RGB frame domains to optimize nighttime traffic object detection jointly. To address the challenge of unbalanced multimodal data fusion, we design a learnable adaptive selection and fusion module. This module performs feature ranking and fusion in the channel dimension, allowing efficient multimodal fusion. Additionally, we construct a novel multi-level feature pyramid network based on multimodal attention fusion. This network extracts potential features to enhance robustness in detecting nighttime traffic objects. Furthermore, we curate a dataset for nighttime traffic scenarios comprising RGB frames and corresponding event streams. Through experiments, we demonstrate that our proposed method outperforms current state-of-the-art techniques in event-based, frame-based, and event and frame fusion methods. This highlights the effectiveness of integrating the event and frame domains in enhancing nighttime traffic object detection. |
| format | Article |
| id | doaj-art-79a2dc234a7f4dedbef5e9cdf10ea6ff |
| institution | DOAJ |
| issn | 2667-3258 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | KeAi Communications Co. Ltd. |
| record_format | Article |
| series | Fundamental Research |
| spelling | doaj-art-79a2dc234a7f4dedbef5e9cdf10ea6ff2025-08-20T02:48:43ZengKeAi Communications Co. Ltd.Fundamental Research2667-32582025-07-01541633164410.1016/j.fmre.2023.08.004Nighttime traffic object detection via adaptively integrating event and frame domainsYu Jiang0Yuehang Wang1Minghao Zhao2Yongji Zhang3Hong Qi4The College of Computer Science and Technology, Jilin University, Changchun 130012, China; The Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China; Corresponding author.The College of Computer Science and Technology, Jilin University, Changchun 130012, China; The Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, ChinaThe College of Earth Sciences, Jilin University, Changchun 130012, ChinaThe College of Computer Science and Technology, Jilin University, Changchun 130012, China; The Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, ChinaThe College of Computer Science and Technology, Jilin University, Changchun 130012, China; The Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, ChinaIntelligent perception is crucial in Intelligent Transportation Systems (ITS), with vision cameras as critical components. However, traditional RGB cameras exhibit a significant decline in performance when capturing nighttime traffic scenes, limiting their effectiveness in supporting ITS. In contrast, event cameras possess a high dynamic range (140 dB vs. 60 dB for traditional cameras), enabling them to overcome frame degradation in low-light conditions. Recently, multimodal learning paradigms have made substantial progress in various vision tasks, such as image-text retrieval. Motivated by this progress, we propose an adaptive selection and fusion detection method that leverages both event and RGB frame domains to optimize nighttime traffic object detection jointly. To address the challenge of unbalanced multimodal data fusion, we design a learnable adaptive selection and fusion module. This module performs feature ranking and fusion in the channel dimension, allowing efficient multimodal fusion. Additionally, we construct a novel multi-level feature pyramid network based on multimodal attention fusion. This network extracts potential features to enhance robustness in detecting nighttime traffic objects. Furthermore, we curate a dataset for nighttime traffic scenarios comprising RGB frames and corresponding event streams. Through experiments, we demonstrate that our proposed method outperforms current state-of-the-art techniques in event-based, frame-based, and event and frame fusion methods. This highlights the effectiveness of integrating the event and frame domains in enhancing nighttime traffic object detection.http://www.sciencedirect.com/science/article/pii/S2667325823002376Environmental perceptionIntelligent transportationNighttime object detectionEvent cameraMultimodal learning |
| spellingShingle | Yu Jiang Yuehang Wang Minghao Zhao Yongji Zhang Hong Qi Nighttime traffic object detection via adaptively integrating event and frame domains Fundamental Research Environmental perception Intelligent transportation Nighttime object detection Event camera Multimodal learning |
| title | Nighttime traffic object detection via adaptively integrating event and frame domains |
| title_full | Nighttime traffic object detection via adaptively integrating event and frame domains |
| title_fullStr | Nighttime traffic object detection via adaptively integrating event and frame domains |
| title_full_unstemmed | Nighttime traffic object detection via adaptively integrating event and frame domains |
| title_short | Nighttime traffic object detection via adaptively integrating event and frame domains |
| title_sort | nighttime traffic object detection via adaptively integrating event and frame domains |
| topic | Environmental perception Intelligent transportation Nighttime object detection Event camera Multimodal learning |
| url | http://www.sciencedirect.com/science/article/pii/S2667325823002376 |
| work_keys_str_mv | AT yujiang nighttimetrafficobjectdetectionviaadaptivelyintegratingeventandframedomains AT yuehangwang nighttimetrafficobjectdetectionviaadaptivelyintegratingeventandframedomains AT minghaozhao nighttimetrafficobjectdetectionviaadaptivelyintegratingeventandframedomains AT yongjizhang nighttimetrafficobjectdetectionviaadaptivelyintegratingeventandframedomains AT hongqi nighttimetrafficobjectdetectionviaadaptivelyintegratingeventandframedomains |