The invention provides an automatic driving method and system based on multi-view and multi-modal fusion, and relates to the technical field of automatic driving. Multi-view and multi-modal observation data are processed through an automatic driving perception network, and a global feature vector and a brake score are generated; inputting the global feature vector, the current driving position of the vehicle and the target position into a waypoint prediction network to obtain predicted waypoints; on the basis of the predicted path point, the vehicle speed information and the brake score, executing control over the vehicle is completed; according to the invention, through an automatic driving perception network composed of four modules including a multi-view semantic segmentation module, a point cloud semantic segmentation module, a multi-modal feature fusion module and an over-distance camera module, the traffic environment around a vehicle is comprehensively perceived; the problems of missing detection of obstacles on the two sides of a vehicle body and illegal behaviors caused by the fact that the traffic lights are not correctly recognized are solved, and understanding of behaviors of a plurality of traffic participants is enhanced.

    本发明提出了基于多视角多模态融合的自动驾驶方法及系统,涉及自动驾驶技术领域,通过自动驾驶感知网络处理多视角多模态的观测数据,生成全局特征向量和刹车分数;将全局特征向量、车辆行驶的当前位置和目标位置输入到路径点预测网络中,得到预测的路径点;基于预测的路径点、车辆速度信息和刹车分数,完成对车辆的执行控制;本发明通过由多视角语义分割模块、点云语义分割模块、多模态特征融合模块和超距相机模块四个模块组成的自动驾驶感知网络,对车辆周围交通环境进行全面感知,解决了车辆对车身两侧障碍物的漏检测和因没有正确识别交通信号灯而产生的违规行为,加强了对多个交通参与者行为的理解。


    Access

    Download


    Export, share and cite



    Title :

    Automatic driving method and system based on multi-view multi-modal fusion


    Additional title:

    基于多视角多模态融合的自动驾驶方法及系统


    Contributors:
    LIU CHUNSHENG (author) / SUN JINYING (author) / CHANG FALIANG (author) / HUANG YIMING (author) / HAO PENGHUI (author) / LIU HUI (author) / HUAN HENGQIANG (author)

    Publication date :

    2024-01-09


    Type of media :

    Patent


    Type of material :

    Electronic Resource


    Language :

    Chinese


    Classification :

    IPC:    G06V / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G01S RADIO DIRECTION-FINDING , Funkpeilung / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen



    Automatic driving safety assisting method and system based on multi-modal fusion

    CHEN LIN / LIANG HONGBIN / SHANG MINGSHENG et al. | European Patent Office | 2025

    Free access

    End-to-end automatic driving multi-modal data fusion method and system

    SU XIAOCONG / LONG KUN / CHEN JUEXUAN et al. | European Patent Office | 2025

    Free access

    Multi-modal end-to-end automatic driving method and system based on unified aerial view representation

    TAN XIAOJUN / WEI YUZHEN / YU ZE et al. | European Patent Office | 2024

    Free access

    Automatic driving model based on multi-modal large model, training method and automatic driving method

    HUANG JIZHOU / ZENG ZENGFENG | European Patent Office | 2025

    Free access

    Multi-Modal Fusion for Enhanced Automatic Modulation Classification

    Li, Yingkai / Wang, Shufei / Zhang, Yibin et al. | IEEE | 2024