Multi-modal fusion is a promising approach to boost the autonomous driving performance and has already received a large amount of attention. Meanwhile, to increase driving reliability under distinct scenarios, it is important to handle unforeseen weather events in the training dataset, which is known as an Out-Of-Distribution (OOD) problem, for autonomous driving algorithms. In this paper, we consider those two aspects and propose an end-to-end multi-modal domain-enhanced framework, namely CrossFuser, to meet the safety orientated driving requirements. CrossFuser first integrates both image and lidar modalities to generate a robust environmental representation through conjoint mapping, elastic disentanglement, and attention mechanism. Further, the perception embedding is used to calculate corresponding waypoints by a waypoint prediction network, consisting of Gate Recurrent Units (GRUs). Finally, the final control commands are calculated by low-level control functions. We conduct experiments on the Car Learning to Act (CARLA) driving simulator involving complex weather conditions under urban scenarios, the results show that CrossFuser can outperform the state of the art.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    CrossFuser: Multi-Modal Feature Fusion for End-to-End Autonomous Driving Under Unseen Weather Conditions


    Contributors:
    Wu, Weishang (author) / Deng, Xiaoheng (author) / Jiang, Ping (author) / Wan, Shaohua (author) / Guo, Yuanxiong (author)

    Published in:

    Publication date :

    2023-12-01


    Size :

    5779457 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Benchmarking Image Sensors Under Adverse Weather Conditions for Autonomous Driving

    Bijelic, Mario / Gruber, Tobias / Ritter, Werner | IEEE | 2018




    EVALUATING MULTI-MODAL TRAJECTORY PREDICTIONS FOR AUTONOMOUS DRIVING

    ETTINGER SCOTT MORGAN / ANGUELOV DRAGOMIR | European Patent Office | 2022

    Free access

    Multi-sensor fusion for autonomous driving

    Zhang, Xinyu / Li, Jun / Li, Zhiwei et al. | TIBKAT | 2023