The invention provides an automatic driving collaborative perception method and device based on a multi-modal large model, and the method comprises the steps: carrying out the processing of the point cloud data of a main-end vehicle through the multi-modal large model, and obtaining text information; text features are extracted from the text information, image features are extracted from the image data of the main end vehicle, and depth map features are extracted from a depth map corresponding to the point cloud data; fusing the depth map features and the image features according to the text features to obtain first fusion features; fusing the first fusion feature and a to-be-detected object feature sent by the target end to obtain a second fusion feature; the target end comprises at least one of the cooperation end and the road end of the main end vehicle; and executing a multi-terminal collaborative perception visual task based on the second fusion feature. According to the method provided by the invention, the characterization capability of the sensing features is improved, and the collaborative sensing accuracy and robustness among the multi-terminal vehicles are further improved.

    本发明提供一种基于多模态大模型的自动驾驶协同感知方法及装置,该方法包括:通过多模态大模型对主端车辆的点云数据进行处理,得到文本信息;从文本信息中提取文本特征,从主端车辆的图像数据中提取图像特征,从点云数据对应的深度图中提取深度图特征;根据文本特征对深度图特征和图像特征进行融合,得到第一融合特征;对第一融合特征和目标端发送的待检测物体特征进行融合,得到第二融合特征;目标端包括主端车辆的协同端和路端中的至少一项;基于第二融合特征执行多端协同感知视觉任务。本发明所述方法提高了感知特征的表征能力,进而提高了多终端车辆间的协同感知准确率和鲁棒性。


    Access

    Download


    Export, share and cite



    Title :

    Automatic driving collaborative perception method and device based on multi-modal large model


    Additional title:

    基于多模态大模型的自动驾驶协同感知方法及装置


    Contributors:
    DAI XINGYUAN (author) / NAN YANGCHEN (author) / ZHAO HONGXIA (author) / GUO CHAO (author) / LYU YISHENG (author)

    Publication date :

    2024-11-01


    Type of media :

    Patent


    Type of material :

    Electronic Resource


    Language :

    Chinese


    Classification :

    IPC:    G06V / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G01D MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE , Anzeigen oder Aufzeichnen in Verbindung mit Messen allgemein / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / G08G Anlagen zur Steuerung, Regelung oder Überwachung des Verkehrs , TRAFFIC CONTROL SYSTEMS



    Automatic driving model based on multi-modal large model, training method and automatic driving method

    HUANG JIZHOU / ZENG ZENGFENG | European Patent Office | 2025

    Free access


    Automatic driving collaborative perception method and system based on vehicle shortage perception enhancement

    DAI PENGLIN / WEI QUANMIN / LI WEI et al. | European Patent Office | 2024

    Free access

    Automatic driving decision-making system based on multi-modal perception and hierarchical action

    ZHENG KAI / SU HAN / LIU SHUNCHENG et al. | European Patent Office | 2023

    Free access

    Automatic driving vehicle dynamic pick-up management system based on multi-modal large model

    LIANG XIAOHUI / CHEN RONGBO / CAI YIJIA et al. | European Patent Office | 2025

    Free access