In this paper, a multimodal user-emotion detection system for social robots is presented. This system is intended to be used during human-robot interaction, and it is integrated as part of the overall interaction system of the robot: the Robotics Dialog System (RDS). Two modes are used to detect emotions: the voice and face expression analysis. In order to analyze the voice of the user, a new component has been developed: Gender and Emotion Voice Analysis (GEVA), which is written using the Chuck language. For emotion detection in facial expressions, the system, Gender and Emotion Facial Analysis (GEFA), has been also developed. This last system integrates two third-party solutions: Sophisticated High-speed Object Recognition Engine (SHORE) and Computer Expression Recognition Toolbox (CERT). Once these new components (GEVA and GEFA) give their results, a decision rule is applied in order to combine the information given by both of them. The result of this rule, the detected emotion, is integrated into the dialog system through communicative acts. Hence, each communicative act gives, among other things, the detected emotion of the user to the RDS so it can adapt its strategy in order to get a greater satisfaction degree during the human-robot dialog. Each of the new components, GEVA and GEFA, can also be used individually. Moreover, they are integrated with the robotic control platform ROS (Robot Operating System). Several experiments with real users were performed to determine the accuracy of each component and to set the final decision rule. The results obtained from applying this decision rule in these experiments show a high success rate in automatic user emotion recognition, improving the results given by the two information channels (audio and visual) separately. ; The authors gratefully acknowledge the funds provided by the Spanish MICINN (Ministry of Science and Innovation) through the project “Aplicaciones de los robots sociales”, DPI2011-26980 from the Spanish Ministry of Economy and Competitiveness. Moreover, the research leading to these results has received funding from the RoboCity2030-II-CM project (S2009/DPI-1559), funded by Programas de Actividades I+D en la Comunidad de Madrid and cofunded by Structural Funds of the EU.


    Access

    Download


    Export, share and cite



    Title :

    A multimodal emotion detection system during human-robot interaction



    Publication date :

    2013-11-14


    Remarks:

    AR/0000014610



    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Classification :

    DDC:    629




    Multimodal Emotion Recognition and Intention Understanding in Human-Robot Interaction

    Chen, Luefeng / Liu, Zhentao / Wu, Min et al. | Springer Verlag | 2021



    Designing a Multimodal Human-Robot Interaction Interface for an Industrial Robot

    Mocan, Bogdan / Fulea, Mircea / Brad, Stelian | Springer Verlag | 2015


    Affective Touch in Human–Robot Interaction : Conveying Emotion to the Nao Robot

    Andreasson, Rebecca / Alenljung, Beatrice / Billing, Erik et al. | BASE | 2018

    Free access

    Affective touch in human–robot interaction : Conveying emotion to the Nao robot

    Andreasson, Rebecca / Alenljung, Beatrice / Billing, Erik et al. | BASE | 2018

    Free access