The embodiment of the invention relates to a multi-mode instruction recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining at least one kind of induction information collected by induction equipment on a vehicle, and obtaining the current vehicle state information of the vehicle; carrying out feature coding on the at least one kind of induction information by utilizing an induction information encoder to obtain induction mode feature data; performing feature coding on the vehicle state information by using a vehicle state encoder to obtain vehicle state modal feature data; fusing the induction modal feature data and the vehicle state modal feature data by using a multi-modal feature adapter to obtain fused feature data; and identifying the fused feature data by using an instruction identification network to obtain instruction information for controlling the vehicle. According to the embodiment of the invention, end-to-end multi-modal information identification is realized, the vehicle state and the sensing information are integrated, the instruction intention of the user can be identified more accurately, and the scheme is relatively high in compatibility and expandability.

    本申请实施例涉及一种多模态指令识别方法、装置、电子设备及存储介质,上述方法包括:获取车辆上的感应设备采集的至少一种感应信息,并获取车辆当前的车辆状态信息;利用感应信息编码器,对至少一种感应信息进行特征编码,得到感应模态特征数据;利用车辆状态编码器,对车辆状态信息进行特征编码,得到车辆状态模态特征数据;利用多模态特征适配器,对感应模态特征数据和车辆状态模态特征数据进行融合,得到融合特征数据;利用指令识别网络,对融合特征数据进行识别,得到用于控制车辆的指令信息。本申请实施例实现了端到端地进行多模态信息识别,综合车辆状态和感应信息,可以更精确地识别用户的指令意图,且方案的兼容性和可扩展性较强。


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Multi-mode instruction identification method and device, electronic equipment and storage medium


    Weitere Titelangaben:

    多模态指令识别方法、装置、电子设备及存储介质


    Beteiligte:
    YU RUI (Autor:in) / JIANG XIAOLING (Autor:in)

    Erscheinungsdatum :

    10.09.2024


    Medientyp :

    Patent


    Format :

    Elektronische Ressource


    Sprache :

    Chinesisch


    Klassifikation :

    IPC:    B60R Fahrzeuge, Fahrzeugausstattung oder Fahrzeugteile, soweit nicht anderweitig vorgesehen , VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR / B60H ARRANGEMENTS OR ADAPTATIONS OF HEATING, COOLING, VENTILATING, OR OTHER AIR-TREATING DEVICES SPECIALLY FOR PASSENGER OR GOODS SPACES OF VEHICLES , Anordnung oder Einbau der Heizung, Kühlung, Lüftung oder anderer Luftbehandlungsvorrichtungen für die Fahrzeugräume für Reisende oder Fracht / E05F DEVICES FOR MOVING WINGS INTO OPEN OR CLOSED POSITION , Vorrichtungen zum Bewegen der Flügel in die Offen- oder Schließstellung



    Vehicle-mounted multi-instruction execution method and device, electronic equipment and storage medium

    FENG MIN / CHEN LIRONG | Europäisches Patentamt | 2023

    Freier Zugriff


    Traveling crane hoisting instruction processing method and device, electronic equipment and storage medium

    LONG BO / WANG LIN / WANG TIANYUN | Europäisches Patentamt | 2024

    Freier Zugriff

    Multi-mode driving state detection method and device, electronic equipment and storage medium

    LUO YUXING / DENG LI / MOU GUANNAN et al. | Europäisches Patentamt | 2024

    Freier Zugriff

    Mode switching method and device, electronic equipment and storage medium

    YIN JIANKUN / MA YANHONG / GUO DINGYI | Europäisches Patentamt | 2023

    Freier Zugriff