The invention belongs to the technical field of emotion recognition, and provides a multi-modal data fusion driver emotion recognition method and device, and the method comprises the steps: obtaining a multi-modal data set from a collected original data set of driver eyes, an original data set of a vehicle, and an original data set of an environment; performing feature extraction on the data in the multi-modal data set to obtain audio features, text features and visual features; performing feature alignment on the audio features, the text features and the visual features to obtain modal alignment features; performing feature fusion on the modal alignment features by using a mixed attention mechanism to obtain modal fusion features; and performing emotion recognition processing on the modal fusion features to obtain an emotion recognition result. According to the method, efficient fusion of the multi-modal data in the feature layer is realized, and the accuracy of driver emotion recognition is improved.
本发明属于情绪识别技术领域,提供了一种多模态数据融合的驾驶员情绪识别方法及装置,该方法包括:从采集的驾驶员眼睛的原始数据集、车辆的原始数据集和环境的原始数据集中获取多模态数据集;对多模态数据集中的数据进行特征提取,得到音频特征、文本特征和视觉特征;将音频特征、文本特征和视觉特征进行特征对齐,得到模态对齐特征;使用混合注意力机制对模态对齐特征进行特征融合,得到模态融合特征;对模态融合特征进行情绪识别处理,得到情绪识别结果。本发明实现了多模态数据在特征层的高效融合,提高了在驾驶员情绪识别方面的准确性。
Driver emotion recognition method based on multi-modal data fusion
一种多模态数据融合的驾驶员情绪识别方法
2025-02-25
Patent
Electronic Resource
Chinese
IPC: | B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / A61B DIAGNOSIS , Diagnostik / G06F ELECTRIC DIGITAL DATA PROCESSING , Elektrische digitale Datenverarbeitung / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / G10L Analyse oder Synthese von Sprache , SPEECH ANALYSIS OR SYNTHESIS |
Speech-Based Driver Emotion Recognition
TIBKAT | 2023
|Speech-Based Driver Emotion Recognition
Springer Verlag | 2022
|Multi-modal data fusion method and device for driver behavior prediction
European Patent Office | 2024
|Driver emotion recognition method and recognition device
European Patent Office | 2024
|Driver emotion recognition method and device based on various data
European Patent Office | 2024
|