The invention provides an automatic driving method and system based on multi-view and multi-modal fusion, and relates to the technical field of automatic driving. Multi-view and multi-modal observation data are processed through an automatic driving perception network, and a global feature vector and a brake score are generated; inputting the global feature vector, the current driving position of the vehicle and the target position into a waypoint prediction network to obtain predicted waypoints; on the basis of the predicted path point, the vehicle speed information and the brake score, executing control over the vehicle is completed; according to the invention, through an automatic driving perception network composed of four modules including a multi-view semantic segmentation module, a point cloud semantic segmentation module, a multi-modal feature fusion module and an over-distance camera module, the traffic environment around a vehicle is comprehensively perceived; the problems of missing detection of obstacles on the two sides of a vehicle body and illegal behaviors caused by the fact that the traffic lights are not correctly recognized are solved, and understanding of behaviors of a plurality of traffic participants is enhanced.
本发明提出了基于多视角多模态融合的自动驾驶方法及系统,涉及自动驾驶技术领域,通过自动驾驶感知网络处理多视角多模态的观测数据,生成全局特征向量和刹车分数;将全局特征向量、车辆行驶的当前位置和目标位置输入到路径点预测网络中,得到预测的路径点;基于预测的路径点、车辆速度信息和刹车分数,完成对车辆的执行控制;本发明通过由多视角语义分割模块、点云语义分割模块、多模态特征融合模块和超距相机模块四个模块组成的自动驾驶感知网络,对车辆周围交通环境进行全面感知,解决了车辆对车身两侧障碍物的漏检测和因没有正确识别交通信号灯而产生的违规行为,加强了对多个交通参与者行为的理解。
Automatic driving method and system based on multi-view multi-modal fusion
基于多视角多模态融合的自动驾驶方法及系统
2024-01-09
Patent
Elektronische Ressource
Chinesisch
IPC: | G06V / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G01S RADIO DIRECTION-FINDING , Funkpeilung / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen |
MULTI-MODAL MOVEMENT PLAN FRAMEWORK FOR AUTOMATIC DRIVING VEHICLE
Europäisches Patentamt | 2020
|Automatic driving decision-making system based on multi-modal perception and hierarchical action
Europäisches Patentamt | 2023
|Multi-sensory and Multi-modal Fusion for Sentient Computing
British Library Online Contents | 2007
|Semantic-Aware Multi-modal Sensor Fusion for Motion Planning in Autonomous Driving
Springer Verlag | 2022
|