The invention discloses a multi-modal end-to-end automatic driving method and system based on unified aerial view representation, and the method comprises the steps: obtaining an RGB camera image and LiDAR point cloud data of a target vehicle object, carrying out the feature extraction through a dual-branch network, converting the image features and LiDAR point cloud features into an aerial view space, and obtaining multi-modal aerial view features; the multi-modal aerial view features are aligned in the channel dimension, and multi-modal aerial view feature fusion is carried out through a convolution self-attention mechanism fusion module; and adopting a behavior planning network based on a Transform decoder, taking the fused aerial view features as decision input of the behavior planning network, and outputting an automatic driving prediction behavior of the target vehicle object. The method effectively improves the environment scene understanding capability and planning performance of an end-to-end automatic driving system, improves the accuracy of automatic driving track prediction, and can be widely applied to the technical field of automatic driving.
本发明公开了一种基于统一鸟瞰图表征的多模态端到端自动驾驶方法及系统,方法包括:获取目标车辆对象的RGB相机图像和LiDAR点云数据,使用双分支网络进行特征提取,并将图像特征和LiDAR点云特征转化到鸟瞰图空间下,得到多模态鸟瞰图特征;将多模态鸟瞰图特征在通道维度对齐,通过卷积自注意力机制融合模块进行多模态鸟瞰图特征融合;采用基于Transformer解码器的行为规划网络,将融合后的鸟瞰图特征作为行为规划网络的决策输入,输出目标车辆对象的自动驾驶预测行为。本发明有效地提升端到端自动驾驶系统的环境场景理解能力和规划性能,提升了自动驾驶轨迹预测的准确性,可广泛应用于自动驾驶技术领域。
Multi-modal end-to-end automatic driving method and system based on unified aerial view representation
基于统一鸟瞰图表征的多模态端到端自动驾驶方法及系统
2024-11-29
Patent
Electronic Resource
Chinese
IPC: | G06V / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen |
Automatic driving method and system based on multi-view multi-modal fusion
European Patent Office | 2024
|European Patent Office | 2025
|Automatic driving 3D operation scene generation method based on aerial view perception
European Patent Office | 2025
|Automatic driving safety assisting method and system based on multi-modal fusion
European Patent Office | 2025
|End-to-end automatic driving multi-modal data fusion method and system
European Patent Office | 2025
|