The invention provides a multi-sensory vehicle-mounted interaction method and system based on multi-modal analysis. The method comprises the following steps: capturing and storing multi-modal source data of a driver in real time through a vehicle-mounted camera, a microphone and a temperature and humidity sensor; analyzing the multi-modal source data of the driver in real time, and extracting feature vectors in the multi-modal source data; splicing the plurality of feature vectors and converting the plurality of feature vectors to the same dimension to obtain processed data; inputting the processed data into a BP neural network for training, and judging the real-time state of the driver; and according to the real-time state of the driver, a corresponding interaction service is actively provided for the driver. By adopting a multi-modal information processing means, the real-time state of a driver can be comprehensively judged, active interaction service is provided, and the accuracy of understanding the emotion and intention of a user is improved; and self-optimization of the vehicle-mounted interaction system is realized through a BP neural network training model.

    本发明提供了一种基于多模态分析的多感官车载交互方法及系统,包括:通过车载摄像头、麦克风、温湿度传感器实时捕获并存储驾驶员的多模态源数据;实时解析所述驾驶员的多模态源数据,提取多模态源数据中的特征向量;将多个特征向量进行拼接并转换到同一维度,得到处理后数据;将处理后数据输入BP神经网络进行训练,判定驾驶员的实时状态;根据驾驶员的实时状态,主动为驾驶员提供相应的交互服务。本发明通过采用多模态信息处理手段,可以综合判断驾驶员实时状态并提供主动式交互服务,提高理解用户情绪、意图的准确度;通过BP神经网络训练模型,实现车载交互系统的自我优化。


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Multi-sensory vehicle-mounted interaction method and system based on multi-modal analysis


    Weitere Titelangaben:

    基于多模态分析的多感官车载交互方法及系统


    Beteiligte:
    FENG JIE (Autor:in) / ZHANG JUNWEI (Autor:in) / SUN XUEWEN (Autor:in) / ZHANG XINGGUO (Autor:in) / DONG ZHANXUN (Autor:in) / LI YAHONG (Autor:in)

    Erscheinungsdatum :

    2022-05-27


    Medientyp :

    Patent


    Format :

    Elektronische Ressource


    Sprache :

    Chinesisch


    Klassifikation :

    IPC:    B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G06K Erkennen von Daten , RECOGNITION OF DATA / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / G06V



    Multi-sensory and Multi-modal Fusion for Sentient Computing

    Town, C. | British Library Online Contents | 2007


    Human-vehicle interaction system based on multi-sensory emotion regulation

    ZHOU TAO | Europäisches Patentamt | 2023

    Freier Zugriff

    Multi-modal vehicle

    ZHU JIANCHAO | Europäisches Patentamt | 2015

    Freier Zugriff

    Multi-sensory interaction vehicle entertainment method and device

    JI CONGCONG / LIU YUXIANG / WANG XIAOCHEN et al. | Europäisches Patentamt | 2023

    Freier Zugriff

    MULTI-MODAL VEHICLE

    KARADIA NARENDRA HIRALAL | Europäisches Patentamt | 2022

    Freier Zugriff