Novelty Detection in Autonomous Driving: A Generative Multi-Modal Sensor Fusion Approach
This paper presents a bio-inspired, generative Multi-Modal Sensor Fusion (MSF) framework to effectively detecting novel and dynamic situations in the surroundings of Autonomous Vehicle (AV). The MSF framework fuses both proprioceptive (wheel odometry) and exteroceptive (LiDAR point-clouds) sensory i...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Open Journal of Intelligent Transportation Systems |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/11037519/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849420551122059264 |
|---|---|
| author | Hafsa Iqbal Haleema Sadia Abdulla Al-Kaff Fernando Garcie |
| author_facet | Hafsa Iqbal Haleema Sadia Abdulla Al-Kaff Fernando Garcie |
| author_sort | Hafsa Iqbal |
| collection | DOAJ |
| description | This paper presents a bio-inspired, generative Multi-Modal Sensor Fusion (MSF) framework to effectively detecting novel and dynamic situations in the surroundings of Autonomous Vehicle (AV). The MSF framework fuses both proprioceptive (wheel odometry) and exteroceptive (LiDAR point-clouds) sensory inputs. A novel 3-Dimensional Dynamic Variational Auto-Encoder (3D-DVAE) model is employed to learn attention-focused distributions from point-clouds in an unsupervised manner. By fusing the distributions of both modalities (wheel and lidar), modality-specific experts’ distributions are learned, capturing both proprioceptive and exteroceptive information from the surroundings. Bayesian Filtering is then applied to detect novel situations/dynamics by probabilistically inferring future states. The proposed method is validated using the KITTI dataset across diverse and complex urban environments. Both quantitative and qualitative results demonstrate the effectiveness of the proposed approach in detecting novelties through multi-modal fusion. |
| format | Article |
| id | doaj-art-ebaa0562fca143889507ff548a6ecbfe |
| institution | Kabale University |
| issn | 2687-7813 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Intelligent Transportation Systems |
| spelling | doaj-art-ebaa0562fca143889507ff548a6ecbfe2025-08-20T03:31:44ZengIEEEIEEE Open Journal of Intelligent Transportation Systems2687-78132025-01-01679981210.1109/OJITS.2025.358027111037519Novelty Detection in Autonomous Driving: A Generative Multi-Modal Sensor Fusion ApproachHafsa Iqbal0https://orcid.org/0000-0002-1773-8066Haleema Sadia1https://orcid.org/0000-0001-8823-2348Abdulla Al-Kaff2https://orcid.org/0000-0003-0212-6400Fernando Garcie3https://orcid.org/0000-0002-6291-5009Departamento de Ingenieria de Sistemas y Automática, Autonomous Mobility and Perception Laboratory, Universidad Carlos III de Madrid, Madrid, SpainDepartment of Electrical and Communication Engineering, United Arab Emirates University, Al Ain, UAEDepartamento de Ingenieria de Sistemas y Automática, Autonomous Mobility and Perception Laboratory, Universidad Carlos III de Madrid, Madrid, SpainDepartamento de Ingenieria de Sistemas y Automática, Autonomous Mobility and Perception Laboratory, Universidad Carlos III de Madrid, Madrid, SpainThis paper presents a bio-inspired, generative Multi-Modal Sensor Fusion (MSF) framework to effectively detecting novel and dynamic situations in the surroundings of Autonomous Vehicle (AV). The MSF framework fuses both proprioceptive (wheel odometry) and exteroceptive (LiDAR point-clouds) sensory inputs. A novel 3-Dimensional Dynamic Variational Auto-Encoder (3D-DVAE) model is employed to learn attention-focused distributions from point-clouds in an unsupervised manner. By fusing the distributions of both modalities (wheel and lidar), modality-specific experts’ distributions are learned, capturing both proprioceptive and exteroceptive information from the surroundings. Bayesian Filtering is then applied to detect novel situations/dynamics by probabilistically inferring future states. The proposed method is validated using the KITTI dataset across diverse and complex urban environments. Both quantitative and qualitative results demonstrate the effectiveness of the proposed approach in detecting novelties through multi-modal fusion.https://ieeexplore.ieee.org/document/11037519/Generative multi-modal sensor fusion frameworkautonomous vehiclesexteroceptive and proprioceptive sensory inputsnovelty detectionBayesian filtering |
| spellingShingle | Hafsa Iqbal Haleema Sadia Abdulla Al-Kaff Fernando Garcie Novelty Detection in Autonomous Driving: A Generative Multi-Modal Sensor Fusion Approach IEEE Open Journal of Intelligent Transportation Systems Generative multi-modal sensor fusion framework autonomous vehicles exteroceptive and proprioceptive sensory inputs novelty detection Bayesian filtering |
| title | Novelty Detection in Autonomous Driving: A Generative Multi-Modal Sensor Fusion Approach |
| title_full | Novelty Detection in Autonomous Driving: A Generative Multi-Modal Sensor Fusion Approach |
| title_fullStr | Novelty Detection in Autonomous Driving: A Generative Multi-Modal Sensor Fusion Approach |
| title_full_unstemmed | Novelty Detection in Autonomous Driving: A Generative Multi-Modal Sensor Fusion Approach |
| title_short | Novelty Detection in Autonomous Driving: A Generative Multi-Modal Sensor Fusion Approach |
| title_sort | novelty detection in autonomous driving a generative multi modal sensor fusion approach |
| topic | Generative multi-modal sensor fusion framework autonomous vehicles exteroceptive and proprioceptive sensory inputs novelty detection Bayesian filtering |
| url | https://ieeexplore.ieee.org/document/11037519/ |
| work_keys_str_mv | AT hafsaiqbal noveltydetectioninautonomousdrivingagenerativemultimodalsensorfusionapproach AT haleemasadia noveltydetectioninautonomousdrivingagenerativemultimodalsensorfusionapproach AT abdullaalkaff noveltydetectioninautonomousdrivingagenerativemultimodalsensorfusionapproach AT fernandogarcie noveltydetectioninautonomousdrivingagenerativemultimodalsensorfusionapproach |