We present an approach to detecting and recognizing spoken isolated phrases based solely on visual input. We adopt an architecture that first employs discriminative detection of visual speech and articulate features, and then performs recognition using a model that accounts for the loose synchronization of the feature streams. Discriminative classifiers detect the subclass of lip appearance corresponding to the presence of speech, and further decompose it into features corresponding to the physical components of articulate production. These components often evolve in a semi-independent fashion, and conventional viseme-based approaches to recognition fail to capture the resulting co-articulation effects. We present a novel dynamic Bayesian network with a multi-stream structure and observations consisting of articulate feature classifier scores, which can model varying degrees of co-articulation in a principled way. We evaluate our visual-only recognition system on a command utterance task. We show comparative results on lip detection and speech/non-speech classification, as well as recognition performance against several baseline systems.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Visual speech recognition with loosely synchronized feature streams


    Contributors:
    Saenko, K. (author) / Livescu, K. (author) / Siracusa, M. (author) / Wilson, K. (author) / Glass, J. (author) / Darrell, T. (author)


    Publication date :

    2005-01-01


    Size :

    198526 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Visual Speech Recognition with Loosely Synchronized Feature Streams

    Saenko, K. / Livescu, K. / Siracusa, M. et al. | British Library Conference Proceedings | 2005


    Dynamic Depth Recovery from Multiple Synchronized Video Streams

    Tao, H. / Sawhney, H. S. / Kumar, R. et al. | British Library Conference Proceedings | 2001


    Dynamic depth recovery from multiple synchronized video streams

    Hai Tao, / Sawhney, H.S. / Kumar, R. | IEEE | 2001


    Wipes, Cotton, Loosely Woven

    SAE Technical Papers | 2016


    WIPES, COTTON, LOOSELY WOVEN

    SAE Technical Papers | 2011