The invention discloses a multi-modal end-to-end automatic driving method and system based on unified aerial view representation, and the method comprises the steps: obtaining an RGB camera image and LiDAR point cloud data of a target vehicle object, carrying out the feature extraction through a dual-branch network, converting the image features and LiDAR point cloud features into an aerial view space, and obtaining multi-modal aerial view features; the multi-modal aerial view features are aligned in the channel dimension, and multi-modal aerial view feature fusion is carried out through a convolution self-attention mechanism fusion module; and adopting a behavior planning network based on a Transform decoder, taking the fused aerial view features as decision input of the behavior planning network, and outputting an automatic driving prediction behavior of the target vehicle object. The method effectively improves the environment scene understanding capability and planning performance of an end-to-end automatic driving system, improves the accuracy of automatic driving track prediction, and can be widely applied to the technical field of automatic driving.

    本发明公开了一种基于统一鸟瞰图表征的多模态端到端自动驾驶方法及系统,方法包括:获取目标车辆对象的RGB相机图像和LiDAR点云数据,使用双分支网络进行特征提取,并将图像特征和LiDAR点云特征转化到鸟瞰图空间下,得到多模态鸟瞰图特征;将多模态鸟瞰图特征在通道维度对齐,通过卷积自注意力机制融合模块进行多模态鸟瞰图特征融合;采用基于Transformer解码器的行为规划网络,将融合后的鸟瞰图特征作为行为规划网络的决策输入,输出目标车辆对象的自动驾驶预测行为。本发明有效地提升端到端自动驾驶系统的环境场景理解能力和规划性能,提升了自动驾驶轨迹预测的准确性,可广泛应用于自动驾驶技术领域。


    Access

    Download


    Export, share and cite



    Title :

    Multi-modal end-to-end automatic driving method and system based on unified aerial view representation


    Additional title:

    基于统一鸟瞰图表征的多模态端到端自动驾驶方法及系统


    Contributors:
    TAN XIAOJUN (author) / WEI YUZHEN (author) / YU ZE (author) / LYU YUANDONG (author) / SHI BODONG (author) / RUAN YIHAO (author)

    Publication date :

    2024-11-29


    Type of media :

    Patent


    Type of material :

    Electronic Resource


    Language :

    Chinese


    Classification :

    IPC:    G06V / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen



    Automatic driving method and system based on multi-view multi-modal fusion

    LIU CHUNSHENG / SUN JINYING / CHANG FALIANG et al. | European Patent Office | 2024

    Free access

    Automatic driving model based on multi-modal large model, training method and automatic driving method

    HUANG JIZHOU / ZENG ZENGFENG | European Patent Office | 2025

    Free access

    Automatic driving 3D operation scene generation method based on aerial view perception

    RAN QI / LIANG CI / ZHENG WEI et al. | European Patent Office | 2025

    Free access

    Automatic driving safety assisting method and system based on multi-modal fusion

    CHEN LIN / LIANG HONGBIN / SHANG MINGSHENG et al. | European Patent Office | 2025

    Free access

    End-to-end automatic driving multi-modal data fusion method and system

    SU XIAOCONG / LONG KUN / CHEN JUEXUAN et al. | European Patent Office | 2025

    Free access