The invention provides a multi-sensory vehicle-mounted interaction method and system based on multi-modal analysis. The method comprises the following steps: capturing and storing multi-modal source data of a driver in real time through a vehicle-mounted camera, a microphone and a temperature and humidity sensor; analyzing the multi-modal source data of the driver in real time, and extracting feature vectors in the multi-modal source data; splicing the plurality of feature vectors and converting the plurality of feature vectors to the same dimension to obtain processed data; inputting the processed data into a BP neural network for training, and judging the real-time state of the driver; and according to the real-time state of the driver, a corresponding interaction service is actively provided for the driver. By adopting a multi-modal information processing means, the real-time state of a driver can be comprehensively judged, active interaction service is provided, and the accuracy of understanding the emotion and intention of a user is improved; and self-optimization of the vehicle-mounted interaction system is realized through a BP neural network training model.

    本发明提供了一种基于多模态分析的多感官车载交互方法及系统,包括:通过车载摄像头、麦克风、温湿度传感器实时捕获并存储驾驶员的多模态源数据;实时解析所述驾驶员的多模态源数据,提取多模态源数据中的特征向量;将多个特征向量进行拼接并转换到同一维度,得到处理后数据;将处理后数据输入BP神经网络进行训练,判定驾驶员的实时状态;根据驾驶员的实时状态,主动为驾驶员提供相应的交互服务。本发明通过采用多模态信息处理手段,可以综合判断驾驶员实时状态并提供主动式交互服务,提高理解用户情绪、意图的准确度;通过BP神经网络训练模型,实现车载交互系统的自我优化。


    Access

    Download


    Export, share and cite



    Title :

    Multi-sensory vehicle-mounted interaction method and system based on multi-modal analysis


    Additional title:

    基于多模态分析的多感官车载交互方法及系统


    Contributors:
    FENG JIE (author) / ZHANG JUNWEI (author) / SUN XUEWEN (author) / ZHANG XINGGUO (author) / DONG ZHANXUN (author) / LI YAHONG (author)

    Publication date :

    2022-05-27


    Type of media :

    Patent


    Type of material :

    Electronic Resource


    Language :

    Chinese


    Classification :

    IPC:    B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G06K Erkennen von Daten , RECOGNITION OF DATA / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / G06V



    STEERING WHEEL MOUNTED MULTI SENSORY INTERFACE

    GENTNER ALEXANDRE / MATHOT JEAN-CHRISTOPHE | European Patent Office | 2025

    Free access

    Human-vehicle interaction system based on multi-sensory emotion regulation

    ZHOU TAO | European Patent Office | 2023

    Free access

    Multi-sensory interaction vehicle entertainment method and device

    JI CONGCONG / LIU YUXIANG / WANG XIAOCHEN et al. | European Patent Office | 2023

    Free access

    VEHICLE-MOUNTED MULTI-SCREEN INTERACTION METHOD, VEHICLE-MOUNTED CONTROLLER, AUTOMOBILE, AND STORAGE MEDIUM

    ZHANG JIANDE / LUO YU / WEN YONGSHENG et al. | European Patent Office | 2025

    Free access

    Vehicle-mounted multi-screen interaction method, storage medium and vehicle

    JIANG ZHUSHENG / YANG DONGSHENG / TANG MINXIN et al. | European Patent Office | 2024

    Free access