The invention provides an automatic driving model based on a multi-modal large model, a training method and an automatic driving method, and relates to the technical field of computers, in particular to the technical field of automatic driving and artificial intelligence. According to the implementation scheme, a training corpus data set is obtained, wherein the training corpus data set at least comprises a visual text alignment corpus and a spatial understanding training corpus for an automatic driving scene; encoding visual data in the visual text alignment corpus by using a visual encoder to obtain encoded data; mapping the coded data by using a mapping layer; processing the mapped coded data, the mapped text data and the spatial understanding training corpus by using a generation layer to obtain a first prediction result and a second prediction result of the automatic driving model; parameters of the automatic driving model are adjusted based at least on the first prediction result and the second prediction result. The automatic driving model obtained through training in the embodiment of the invention has the understanding ability of multi-modal information and also has the reasoning ability in an automatic driving scene.
本公开提供了一种基于多模态大模型的自动驾驶模型、训练和自动驾驶方法,涉及计算机技术领域,尤其涉及自动驾驶和人工智能技术领域。实现方案为:获取训练语料数据集,至少包括视觉文本对齐语料和用于自动驾驶场景的空间理解训练语料;利用视觉编码器对视觉文本对齐语料中的视觉数据进行编码,以得到编码数据;利用映射层对编码数据进行映射;利用生成层分别对经映射的编码数据和文本数据以及空间理解训练语料进行处理,以得到自动驾驶模型的第一预测结果和第二预测结果;至少基于第一预测结果和第二预测结果来调整自动驾驶模型的参数。利用本公开的实施例训练得到的自动驾驶模型既具有多模态信息的理解能力,也具有自动驾驶场景下的推理能力。
Automatic driving model based on multi-modal large model, training method and automatic driving method
基于多模态大模型的自动驾驶模型、训练和自动驾驶方法
2025-02-25
Patent
Electronic Resource
Chinese
IPC: | G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G06F ELECTRIC DIGITAL DATA PROCESSING , Elektrische digitale Datenverarbeitung / G06V |
Automatic driving collaborative perception method and device based on multi-modal large model
European Patent Office | 2024
|Distributed training method of automatic driving learning model and automatic driving method
European Patent Office | 2022
|Training method of automatic driving algorithm model, automatic driving method and device
European Patent Office | 2024
|Driving strategy model training method and automatic driving control method
European Patent Office | 2022
|Driving strategy determination method, model training method and automatic driving vehicle
European Patent Office | 2023
|