The invention provides a multi-sensory vehicle-mounted interaction method and system based on multi-modal analysis. The method comprises the following steps: capturing and storing multi-modal source data of a driver in real time through a vehicle-mounted camera, a microphone and a temperature and humidity sensor; analyzing the multi-modal source data of the driver in real time, and extracting feature vectors in the multi-modal source data; splicing the plurality of feature vectors and converting the plurality of feature vectors to the same dimension to obtain processed data; inputting the processed data into a BP neural network for training, and judging the real-time state of the driver; and according to the real-time state of the driver, a corresponding interaction service is actively provided for the driver. By adopting a multi-modal information processing means, the real-time state of a driver can be comprehensively judged, active interaction service is provided, and the accuracy of understanding the emotion and intention of a user is improved; and self-optimization of the vehicle-mounted interaction system is realized through a BP neural network training model.
本发明提供了一种基于多模态分析的多感官车载交互方法及系统,包括:通过车载摄像头、麦克风、温湿度传感器实时捕获并存储驾驶员的多模态源数据;实时解析所述驾驶员的多模态源数据,提取多模态源数据中的特征向量;将多个特征向量进行拼接并转换到同一维度,得到处理后数据;将处理后数据输入BP神经网络进行训练,判定驾驶员的实时状态;根据驾驶员的实时状态,主动为驾驶员提供相应的交互服务。本发明通过采用多模态信息处理手段,可以综合判断驾驶员实时状态并提供主动式交互服务,提高理解用户情绪、意图的准确度;通过BP神经网络训练模型,实现车载交互系统的自我优化。
Multi-sensory vehicle-mounted interaction method and system based on multi-modal analysis
基于多模态分析的多感官车载交互方法及系统
2022-05-27
Patent
Elektronische Ressource
Chinesisch
IPC: | B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G06K Erkennen von Daten , RECOGNITION OF DATA / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / G06V |
Multi-sensory and Multi-modal Fusion for Sentient Computing
British Library Online Contents | 2007
|Human-vehicle interaction system based on multi-sensory emotion regulation
Europäisches Patentamt | 2023
|Multi-sensory interaction vehicle entertainment method and device
Europäisches Patentamt | 2023
|