In indoor places, such as homes or offices, when abnormal events occur, the behavior and voice of individuals or groups will display abnormal signals. These signals can be both visual and auditory, and they interact and complement each other to jointly create a sense of emotional atmosphere within the scene. In order to achieve effective and accurate perception and response of abnormal emotion during the interaction in smart home, a model of abnormal emotion recognition based on audio-visual modality fusion is proposed. Human skeleton motion data and audio data are utilized to construct separate deep learning networks for action recognition and speech emotion recognition. The accuracy rate achieved on the G3D dataset is 100% and the accuracy rate achieved on the CASIA corpus is 90.83%. For decision-level multimodal fusion, the predicted results of actions and speech emotions are mapped to the “abnormal” axis through fuzzification and weighted average methods. In this process, considerations are taken into account for the varying contributions of different speech emotions and behaviors to the abnormal emotion, as well as the recognition recall rates of the unimodal emotion models. Then the two modalities are allowed to mutually modify each other and achieve quantitative analysis of abnormal emotion through weighted additive fusion.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Abnormal Emotion Recognition Based on Audio-Visual Modality Fusion


    Additional title:

    Lect.Notes Computer


    Contributors:
    Yang, Huayong (editor) / Liu, Honghai (editor) / Zou, Jun (editor) / Yin, Zhouping (editor) / Liu, Lianqing (editor) / Yang, Geng (editor) / Ouyang, Xiaoping (editor) / Wang, Zhiyong (editor) / Jiang, Yutong (author) / Hirota, Kaoru (author)

    Conference:

    International Conference on Intelligent Robotics and Applications ; 2023 ; Hangzhou, China July 05, 2023 - July 07, 2023



    Publication date :

    2023-10-21


    Size :

    12 pages




    Type of media :

    Article/Chapter (Book)


    Type of material :

    Electronic Resource


    Language :

    English




    Abnormal Emotion Recognition Based on Audio-Visual Modality Fusion

    Jiang, Yutong / Hirota, Kaoru / Dai, Yaping et al. | TIBKAT | 2023



    Audio-Visual Based Emotion Recognition -- A New Approach

    Song, M. / Bu, J. / Chen, C. et al. | British Library Conference Proceedings | 2004


    Audio-visual based emotion recognition - a new approach

    Mingli Song, / Jiajun Bu, / Chun Chen, et al. | IEEE | 2004


    CMM: LiDAR-Visual Fusion with Cross-Modality Module for Large-Scale Place Recognition

    Li, Bin / Lu, Fan / Chen, Guang et al. | SAE Technical Papers | 2023