The embodiment of the invention relates to a multi-mode instruction recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining at least one kind of induction information collected by induction equipment on a vehicle, and obtaining the current vehicle state information of the vehicle; carrying out feature coding on the at least one kind of induction information by utilizing an induction information encoder to obtain induction mode feature data; performing feature coding on the vehicle state information by using a vehicle state encoder to obtain vehicle state modal feature data; fusing the induction modal feature data and the vehicle state modal feature data by using a multi-modal feature adapter to obtain fused feature data; and identifying the fused feature data by using an instruction identification network to obtain instruction information for controlling the vehicle. According to the embodiment of the invention, end-to-end multi-modal information identification is realized, the vehicle state and the sensing information are integrated, the instruction intention of the user can be identified more accurately, and the scheme is relatively high in compatibility and expandability.
本申请实施例涉及一种多模态指令识别方法、装置、电子设备及存储介质,上述方法包括:获取车辆上的感应设备采集的至少一种感应信息,并获取车辆当前的车辆状态信息;利用感应信息编码器,对至少一种感应信息进行特征编码,得到感应模态特征数据;利用车辆状态编码器,对车辆状态信息进行特征编码,得到车辆状态模态特征数据;利用多模态特征适配器,对感应模态特征数据和车辆状态模态特征数据进行融合,得到融合特征数据;利用指令识别网络,对融合特征数据进行识别,得到用于控制车辆的指令信息。本申请实施例实现了端到端地进行多模态信息识别,综合车辆状态和感应信息,可以更精确地识别用户的指令意图,且方案的兼容性和可扩展性较强。
Multi-mode instruction identification method and device, electronic equipment and storage medium
多模态指令识别方法、装置、电子设备及存储介质
2024-09-10
Patent
Electronic Resource
Chinese
IPC: | B60R Fahrzeuge, Fahrzeugausstattung oder Fahrzeugteile, soweit nicht anderweitig vorgesehen , VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR / B60H ARRANGEMENTS OR ADAPTATIONS OF HEATING, COOLING, VENTILATING, OR OTHER AIR-TREATING DEVICES SPECIALLY FOR PASSENGER OR GOODS SPACES OF VEHICLES , Anordnung oder Einbau der Heizung, Kühlung, Lüftung oder anderer Luftbehandlungsvorrichtungen für die Fahrzeugräume für Reisende oder Fracht / E05F DEVICES FOR MOVING WINGS INTO OPEN OR CLOSED POSITION , Vorrichtungen zum Bewegen der Flügel in die Offen- oder Schließstellung |
European Patent Office | 2023
|European Patent Office | 2023
|European Patent Office | 2024
|Multi-mode driving state detection method and device, electronic equipment and storage medium
European Patent Office | 2024
|Mode switching method and device, electronic equipment and storage medium
European Patent Office | 2025
|