People and animals fuse auditory and visual information to obtain robust perception. A particular benefit of such cross-modal analysis is the ability to localize visual events associated with sound sources. We aim to achieve this using computer-vision aided by a single microphone. Past efforts encountered problems stemming from the huge gap between the dimensions involved and the available data. This has led to solutions suffering from low spatio-temporal resolutions. We present a rigorous analysis of the fundamental problems associated with this task. Then, we present a stable and robust algorithm which overcomes past deficiencies. It grasps dynamic audio-visual events with high spatial resolution, and derives a unique solution. The algorithm effectively detects pixels that are associated with the sound, while filtering out other dynamic pixels. It is based on canonical correlation analysis (CCA), where we remove inherent ill-posedness by exploiting the typical spatial sparsity of audio-visual events. The algorithm is simple and efficient thanks to its reliance on linear programming and is free of user-defined parameters. To quantitatively assess the performance, we devise a localization criterion. The algorithm capabilities were demonstrated in experiments, where it overcame substantial visual distractions and audio noise.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Pixels that sound


    Contributors:
    Kidron, E. (author) / Schechner, Y.Y. (author) / Elad, M. (author)


    Publication date :

    2005-01-01


    Size :

    1248373 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Smart Pixels

    IEEE | 1994


    Space time pixels

    Bampanelos, A.A. | BASE | 2003

    Free access

    Optoelectronic Smart Pixels Monolithically Integrated

    IEEE; Lasers and Electro-Optics Society | British Library Conference Proceedings | 1994


    Optoelectronic Smart Pixels Monallithically Integrated

    Liepold, D. / Kehrli, U. / Thelen, K. et al. | IEEE | 1994


    Imaging systems with verification pixels

    JOHNSON RICHARD SCOTT | European Patent Office | 2015

    Free access