The invention belongs to the technical field of emotion recognition, and provides a multi-modal data fusion driver emotion recognition method and device, and the method comprises the steps: obtaining a multi-modal data set from a collected original data set of driver eyes, an original data set of a vehicle, and an original data set of an environment; performing feature extraction on the data in the multi-modal data set to obtain audio features, text features and visual features; performing feature alignment on the audio features, the text features and the visual features to obtain modal alignment features; performing feature fusion on the modal alignment features by using a mixed attention mechanism to obtain modal fusion features; and performing emotion recognition processing on the modal fusion features to obtain an emotion recognition result. According to the method, efficient fusion of the multi-modal data in the feature layer is realized, and the accuracy of driver emotion recognition is improved.

    本发明属于情绪识别技术领域,提供了一种多模态数据融合的驾驶员情绪识别方法及装置,该方法包括:从采集的驾驶员眼睛的原始数据集、车辆的原始数据集和环境的原始数据集中获取多模态数据集;对多模态数据集中的数据进行特征提取,得到音频特征、文本特征和视觉特征;将音频特征、文本特征和视觉特征进行特征对齐,得到模态对齐特征;使用混合注意力机制对模态对齐特征进行特征融合,得到模态融合特征;对模态融合特征进行情绪识别处理,得到情绪识别结果。本发明实现了多模态数据在特征层的高效融合,提高了在驾驶员情绪识别方面的准确性。


    Access

    Download


    Export, share and cite



    Title :

    Driver emotion recognition method based on multi-modal data fusion


    Additional title:

    一种多模态数据融合的驾驶员情绪识别方法


    Contributors:
    WU CHAO (author) / FENG CHENGJUN (author) / ZHANG XIONG (author) / ZHOU PENG (author) / MIN ZHAOYUAN (author) / GUO KUANYOU (author) / CAO FEI (author) / CHEN ZHENYAN (author) / YAO BO (author) / WANG ZHEN (author)

    Publication date :

    2025-02-25


    Type of media :

    Patent


    Type of material :

    Electronic Resource


    Language :

    Chinese


    Classification :

    IPC:    B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / A61B DIAGNOSIS , Diagnostik / G06F ELECTRIC DIGITAL DATA PROCESSING , Elektrische digitale Datenverarbeitung / G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / G10L Analyse oder Synthese von Sprache , SPEECH ANALYSIS OR SYNTHESIS



    Speech-Based Driver Emotion Recognition

    Tan, Haiqiu / Zhang, Haodong / Shi, Jian et al. | TIBKAT | 2023


    Speech-Based Driver Emotion Recognition

    Tan, Haiqiu / Zhang, Haodong / Shi, Jian et al. | Springer Verlag | 2022


    Multi-modal data fusion method and device for driver behavior prediction

    LI LIJIANG / ZHAO XUE / HU MIN et al. | European Patent Office | 2024

    Free access

    Driver emotion recognition method and recognition device

    LIANG MINGYU / LIU DANQI / LI XIN et al. | European Patent Office | 2024

    Free access

    Driver emotion recognition method and device based on various data

    GAO RUI / FENG JIAWEI / ZHANG JICUN et al. | European Patent Office | 2024

    Free access