In this work, we address fusion of heterogeneous sensor data using wavelet-based summaries of fused self-similarity information from each sensor. The technique we develop is quite general, does not require domain specific knowledge or physical models, and requires no training. Nonetheless, it can perform surprisingly well at the general task of differentiating classes of time-ordered behavior sequences which are sensed by more than one modality. As a demonstration of our capabilities in the audio to video context, we focus on the differentiation of speech sequences. Data from two or more modalities first are represented using self-similarity matrices(SSMs) corresponding to time-ordered point clouds in feature spaces of each of these data sources; we note that these feature spaces can be of entirely different scale and dimensionality. A fused similarity template is then derived from the modality-specific SSMs using a technique called similarity network fusion (SNF). We investigate pipelines using SNF as both an upstream (feature-level) and a downstream (ranking-level) fusion technique. Multiscale geometric features of this template are then extracted using a recently-developed technique called the scattering transform, and these features are then used to differentiate speech sequences. This method outperforms unsupervised techniques which operate directly on the raw data, and it also outperforms stovepiped methods which operate on SSMs separately derived from the distinct modalities. The benefits of this method become even more apparent as the simulated peak signal to noise ratio decreases.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Multi-Scale Geometric Summaries for Similarity-Based Sensor Fusion


    Beteiligte:
    Tralie, Christopher J. (Autor:in) / Bendich, Paul (Autor:in) / Harer, John (Autor:in)

    Erschienen in:

    Erscheinungsdatum :

    01.03.2019


    Format / Umfang :

    3468948 byte




    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Multi-resolution geometric fusion

    Hilton, A. / Illingworth, J. | IEEE | 1997


    Multi-Resolution Geometric Fusion

    Hilton, A. / Illingworth, J. / National Research Council of Canada | British Library Conference Proceedings | 1997


    Image Inpainting Based on Geometric Similarity

    Hung, K.-M. / Chen, Y.-L. / Hsieh, C.-T. | British Library Online Contents | 2012


    Image Inpainting Based On Geometric Similarity

    Hung, Kuo-Ming / Chen, Yen-Liang / Hsieh, Ching-Tang | British Library Online Contents | 2012


    Geometric fusion for a hand-held 3D sensor

    Hilton, A. / Illingworth, J. | British Library Online Contents | 2000