Many learning-based methods estimate ego-motion using visual sensors. However, visual sensors are prone to intense lighting variations and textureless scenarios. 4D radar, an emerging automotive sensor, complements visual sensors effectively due to its robustness in adverse weather and lighting conditions. This paper presents an end-to-end 4D radar-visual odometry (4DRVO) approach that combines sparse point cloud data from 4D radar with image information from cameras. Using the Feature Pyramid, Pose Warping, and Cost Volume (PWC) network architecture, we extract 4D radar point features and image features at multiple scales. We then employ a hierarchical iterative refinement approach to supervise the estimated pose. We propose a novel Cross-Modal Transformer (CMT) module to effectively fuse the 4D radar point modality, image modality, and 4D radar point-image connection modality at multiple scales, achieving cross-modal feature interaction and multi-modal feature fusion. Additionally, we designed a point confidence estimation module to mitigate the impact of dynamic objects on odometry estimation. Extensive experiments were conducted on the View-of-Delft (VoD) dataset, showcasing the remarkable performance and effectiveness of the proposed 4D radar-visual odometry method.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Deep 4D Automotive Radar-Camera Fusion Odometry with Cross-Modal Transformer Fusion


    Additional title:

    Sae Technical Papers


    Contributors:
    Zhuo, Guirong (author) / Xiong, Lu (author) / Zhou, Mingyu (author) / Lu, Shouyi (author) / Lu, Xinfei (author)

    Conference:

    SAE 2023 Intelligent and Connected Vehicles Symposium ; 2023



    Publication date :

    2023-12-20




    Type of media :

    Conference paper


    Type of material :

    Print


    Language :

    English




    4DRVO-Net: Deep 4D Radar–Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion

    Zhuo, Guirong / Lu, Shouyi / Zhou, Huanyu et al. | IEEE | 2024


    Multi-Camera Sensor Fusion for Visual Odometry using Deep Uncertainty Estimation

    Kaygusuz, Nimet / Mendez, Oscar / Bowden, Richard | IEEE | 2021


    Efficient Deep-Learning 4D Automotive Radar Odometry Method

    Lu, Shouyi / Zhuo, Guirong / Xiong, Lu et al. | IEEE | 2024


    LiDAR-Camera Fusion for Depth Enhanced Unsupervised Odometry

    Fetic, Naida / Aydemir, Eren / Unel, Mustafa | IEEE | 2022


    Automotive visual odometry

    Buczko, Martin / Shaker Verlag | TIBKAT | 2018