The invention relates to an automatic driving safety assisting method and system based on multi-modal fusion, and belongs to the field of automatic driving. The system is composed of a multi-mode sensor, a preprocessing unit, a feature encoder, a time sequence merging module, a context attention module, a classification module, a data storage unit and an output unit. The method comprises the following steps: S1, collecting visual image data and non-visual image data; s2, preprocessing the visual image data and the non-visual image data; s3, performing feature extraction by using a feature encoder; s4, key event division is carried out by using a time sequence merging module; S5, feature fusion is carried out by using a context attention module; s6, inputting the fusion features into a classification module, and predicting pedestrian behavior types; and S7, storing a pedestrian behavior type prediction result, and feeding back the pedestrian behavior type prediction result to the automobile terminal system. According to the method, key events related to pedestrian behaviors can be effectively identified and aggregated, and the accuracy and efficiency of pedestrian behavior intention prediction are improved.
本发明涉及基于多模态融合的自动驾驶安全性辅助方法和系统,属于自动驾驶领域。该系统由多模态传感器、预处理单元、特征编码器、时序合并模块、上下文注意力模块、分类模块、数据存储单元、输出单元构成。该方法包含以下步骤:S1:采集视觉图像数据和非视觉图像数据;S2:对视觉图像数据和非视觉图像数据进行预处理;S3:利用特征编码器进行特征提取;S4:利用时序合并模块进行关键事件划分:S5:利用上下文注意力模块对进行特征融合;S6:将融合特征输入分类模块中,预测行人行为类型;S7:将预测行人行为类型结果存储,并反馈给汽车终端系统。本发明方法能够有效地识别和聚合与行人行为相关的关键事件,提高行人行为意图预测的准确性和效率。
Automatic driving safety assisting method and system based on multi-modal fusion
基于多模态融合的自动驾驶安全性辅助方法和系统
2025-02-21
Patent
Electronic Resource
Chinese
IPC: | G06V / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G06F ELECTRIC DIGITAL DATA PROCESSING , Elektrische digitale Datenverarbeitung / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen |
AUTOMATIC DRIVING ASSISTING SYSTEM, AUTOMATIC DRIVING ASSISTING METHOD, AND COMPUTER PROGRAM
European Patent Office | 2015
|AUTOMATIC DRIVING ASSISTING SYSTEM, AUTOMATIC DRIVING ASSISTING METHOD, AND COMPUTER PROGRAM
European Patent Office | 2016
|Automatic driving method and system based on multi-view multi-modal fusion
European Patent Office | 2024
|End-to-end automatic driving multi-modal data fusion method and system
European Patent Office | 2025
|