In this paper we present (1) a processing architecture used to collect multi-modal sensor data, both for corpora collection and real-time processing, (2) an open-source implementation thereof and (3) a use-case where we deploy the architecture in a multi-party deception game, featuring six human players and one robot. The architecture is agnostic to the choice of hardware (e.g. microphones, cameras, etc.) and programming languages, although our implementation is mostly written in Python. In our use-case, different methods of capturing verbal and non-verbal cues from the participants were used. These were processed in real-time and used to inform the robot about the participants’ deceptive behaviour. The framework is of particular interest for researchers who are interested in the collection of multi-party, richly recorded corpora and the design of conversational systems. Moreover for researchers who are interested in human-robot interaction the available modules offer the possibility to easily create both autonomous and wizard-of-Oz interactions. ; QC 20180618


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    FARMI : A Framework for Recording Multi-Modal Interactions


    Beteiligte:

    Erscheinungsdatum :

    2018-01-01


    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Klassifikation :

    DDC:    629



    Modeling Cross-Modal Interaction in a Multi-detector, Multi-modal Tracking Framework

    Zhong, Yiqi / You, Suya / Neumann, Ulrich | British Library Conference Proceedings | 2021


    A Probabilistic Framework for Multi-modal Multi-Person Tracking

    Checka, Neal / Wilson, Kevin / Rangarajan, Vibhav et al. | IEEE | 2003


    An Integrity Assessment Framework for Multi-modal Vehicle Localization

    Balakrishnan, Arjun / Rodriguez F., Sergio A. / Reynaud, Roger | IEEE | 2019


    MULTI-MODAL MOVEMENT PLAN FRAMEWORK FOR AUTOMATIC DRIVING VEHICLE

    ZHANG YAJIA / LI DONG / ZHANG RANRAN et al. | Europäisches Patentamt | 2020

    Freier Zugriff

    DriverSense: A Multi-Modal Framework for Advanced Driver Assistance System

    Vyas, Daiwat Amit / Chaturvedi, Manish | IEEE | 2024