Despite recent advances in gesture recognition, reliance on the visual signal alone to classify unrestricted continuous gesticulation is inherently error-prone. Since spontaneous gesticulation is mostly coverbal in nature, there have been some attempts of using speech cues to improve gesture recognition. Some attempts have been made in using speech cues to improve gesture recognition, e.g., keyword-gesture co-analysis. Use of such scheme is burdened by the complexity of natural language understanding. This paper offers a "signal-level" perspective by exploring prosodic phenomena of spontaneous gesture and speech co-production. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (co-articulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions in multimodal production, are exploited in an audio-visual feature integration framework using hidden Markov models (HMMs). Co-articulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Improving continuous gesture recognition with spoken prosody


    Beteiligte:
    Kettebekov, S. (Autor:in) / Yeasin, M. (Autor:in) / Sharma, R. (Autor:in)


    Erscheinungsdatum :

    01.01.2003


    Format / Umfang :

    990574 byte





    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Improving Continuous Gesture Recognition with Spoken Prosody

    Kettebekov, S. / Yeasin, M. / Sharma, R. et al. | British Library Conference Proceedings | 2003


    Human-Robot Interaction Through Gesture-Free Spoken Dialogue

    Kulyukin, V. | British Library Online Contents | 2004


    Method For a Gesture Recognition System for Improving Gesture Recognition, Device, Computer Program and Vehicle

    FROMMANN THOMAS / HEIJBOER STEFAN | Europäisches Patentamt | 2025

    Freier Zugriff

    GESTURE RECOGNITION DEVICE AND GESTURE RECOGNITION METHOD

    HIGUCHI DAIKI / OHASHI DAISUKE | Europäisches Patentamt | 2022

    Freier Zugriff

    GESTURE RECOGNITION ASSISTANCE APPARATUS AND GESTURE RECOGNITION METHOD

    KIM JEONG / PARK JANG WOONG | Europäisches Patentamt | 2024

    Freier Zugriff