Multi-vehicle cooperative perception has recently emerged for facilitating long-range and large-scale perception ability of connected automated vehicles (CAVs). Nonetheless, enormous efforts formulate collaborative perception as LiDAR-only 3D detection paradigm, neglecting the significance and complementary of dense image. In this work, we construct the first multi-modal vehicle-to-vehicle cooperative perception framework dubbed as V2VFormer++, where individual camera-LiDAR representation is incorporated with dynamic channel fusion (DCF) at bird’s-eye-view (BEV) space and ego-centric BEV maps from adjacent vehicles are aggregated by global-local transformer module. Specifically, channel-token mixer (CTM) with MLP design is developed to capture global response among neighboring CAVs, and position-aware fusion (PAF) further investigate the spatial correlation between each ego-networked map in a local perspective. In this manner, we could strategically determine which CAVs are desirable for collaboration and how to aggregate the foremost information from them. Quantitative and qualitative experiments are conducted on both publicly-available OPV2V and V2X-Sim 2.0 benchmarks, and our proposed V2VFormer++ reports the state-of-the-art cooperative perception performance, demonstrating its effectiveness and advancement. Moreover, ablation study and visualization analysis further suggest the strong robustness against diverse disturbances from real-world scenarios.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    V2VFormer++: Multi-Modal Vehicle-to-Vehicle Cooperative Perception via Global-Local Transformer


    Contributors:
    Yin, Hongbo (author) / Tian, Daxin (author) / Lin, Chunmian (author) / Duan, Xuting (author) / Zhou, Jianshan (author) / Zhao, Dezong (author) / Cao, Dongpu (author)


    Publication date :

    2024-02-01


    Size :

    5031717 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    V2VFormer: Vehicle-to-Vehicle Cooperative Perception With Spatial-Channel Transformer

    Lin, Chunmian / Tian, Daxin / Duan, Xuting et al. | IEEE | 2024


    V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer

    Xu, Runsheng / Xiang, Hao / Tu, Zhengzhong et al. | British Library Conference Proceedings | 2022


    Multi-modal vehicle

    NARENDRA HIRALAL KARADIA | European Patent Office | 2022

    Free access

    MULTI-MODAL VEHICLE

    KARADIA NARENDRA H | European Patent Office | 2022

    Free access

    MULTI-MODAL VEHICLE

    KARADIA NARENDRA HIRALAL | European Patent Office | 2020

    Free access