The invention provides an automatic driving model based on a multi-modal large model, a training method and an automatic driving method, and relates to the technical field of computers, in particular to the technical field of automatic driving and artificial intelligence. According to the implementation scheme, a training corpus data set is obtained, wherein the training corpus data set at least comprises a visual text alignment corpus and a spatial understanding training corpus for an automatic driving scene; encoding visual data in the visual text alignment corpus by using a visual encoder to obtain encoded data; mapping the coded data by using a mapping layer; processing the mapped coded data, the mapped text data and the spatial understanding training corpus by using a generation layer to obtain a first prediction result and a second prediction result of the automatic driving model; parameters of the automatic driving model are adjusted based at least on the first prediction result and the second prediction result. The automatic driving model obtained through training in the embodiment of the invention has the understanding ability of multi-modal information and also has the reasoning ability in an automatic driving scene.

    本公开提供了一种基于多模态大模型的自动驾驶模型、训练和自动驾驶方法,涉及计算机技术领域,尤其涉及自动驾驶和人工智能技术领域。实现方案为:获取训练语料数据集,至少包括视觉文本对齐语料和用于自动驾驶场景的空间理解训练语料;利用视觉编码器对视觉文本对齐语料中的视觉数据进行编码,以得到编码数据;利用映射层对编码数据进行映射;利用生成层分别对经映射的编码数据和文本数据以及空间理解训练语料进行处理,以得到自动驾驶模型的第一预测结果和第二预测结果;至少基于第一预测结果和第二预测结果来调整自动驾驶模型的参数。利用本公开的实施例训练得到的自动驾驶模型既具有多模态信息的理解能力,也具有自动驾驶场景下的推理能力。


    Access

    Download


    Export, share and cite



    Title :

    Automatic driving model based on multi-modal large model, training method and automatic driving method


    Additional title:

    基于多模态大模型的自动驾驶模型、训练和自动驾驶方法


    Contributors:

    Publication date :

    2025-02-25


    Type of media :

    Patent


    Type of material :

    Electronic Resource


    Language :

    Chinese


    Classification :

    IPC:    G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G06F ELECTRIC DIGITAL DATA PROCESSING , Elektrische digitale Datenverarbeitung / G06V



    Automatic driving collaborative perception method and device based on multi-modal large model

    DAI XINGYUAN / NAN YANGCHEN / ZHAO HONGXIA et al. | European Patent Office | 2024

    Free access

    Distributed training method of automatic driving learning model and automatic driving method

    XU XUNNONG / LIU XUAN / SHEN ZHENRUI | European Patent Office | 2022

    Free access

    Training method of automatic driving algorithm model, automatic driving method and device

    WEI KAI / YIN PING / SUN XINGYAN | European Patent Office | 2024

    Free access

    Driving strategy model training method and automatic driving control method

    DENG QI / LI RUYANG / ZHANG YAQIANG et al. | European Patent Office | 2022

    Free access