People and animals fuse auditory and visual information to obtain robust perception. A particular benefit of such cross-modal analysis is the ability to localize visual events associated with sound sources. We aim to achieve this using computer-vision aided by a single microphone. Past efforts encountered problems stemming from the huge gap between the dimensions involved and the available data. This has led to solutions suffering from low spatio-temporal resolutions. We present a rigorous analysis of the fundamental problems associated with this task. Then, we present a stable and robust algorithm which overcomes past deficiencies. It grasps dynamic audio-visual events with high spatial resolution, and derives a unique solution. The algorithm effectively detects pixels that are associated with the sound, while filtering out other dynamic pixels. It is based on canonical correlation analysis (CCA), where we remove inherent ill-posedness by exploiting the typical spatial sparsity of audio-visual events. The algorithm is simple and efficient thanks to its reliance on linear programming and is free of user-defined parameters. To quantitatively assess the performance, we devise a localization criterion. The algorithm capabilities were demonstrated in experiments, where it overcame substantial visual distractions and audio noise.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Pixels that sound


    Beteiligte:
    Kidron, E. (Autor:in) / Schechner, Y.Y. (Autor:in) / Elad, M. (Autor:in)


    Erscheinungsdatum :

    01.01.2005


    Format / Umfang :

    1248373 byte





    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Smart Pixels

    IEEE | 1994


    Space time pixels

    Bampanelos, A.A. | BASE | 2003

    Freier Zugriff

    Optoelectronic Smart Pixels Monolithically Integrated

    IEEE; Lasers and Electro-Optics Society | British Library Conference Proceedings | 1994


    Optoelectronic Smart Pixels Monallithically Integrated

    Liepold, D. / Kehrli, U. / Thelen, K. et al. | IEEE | 1994


    Dynamic demosaicing of camera pixels

    PATEL SHIV G / MURAD MOHANNAD / MACHAK JOSEPH G et al. | Europäisches Patentamt | 2020

    Freier Zugriff