In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first network is a multi modal variational autoencoder that maps directly one modality to the other, while the second one is a single-modal variational autoencoder. In order to associate the two spaces, we apply variational alignment, which acts as a translation mechanism that projects the latent space of the first VAE onto the one of the single-modal VAE through an intermediate distribution. Experimental results on four well-known datasets, covering two different application domains (food image analysis and 3D hand pose estimation), show the generality of the proposed method and its superiority against a number of state-of-the-art approaches.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Cross-modal Variational Alignment of Latent Spaces


    Beteiligte:


    Erscheinungsdatum :

    01.06.2020


    Format / Umfang :

    1009563 byte





    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Cross-Modal Generation and Pair Correlation Alignment Hashing

    Ou, Weihua / Deng, Jiaxin / Zhang, Lei et al. | IEEE | 2023


    Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders

    Zhang, Yingji / Carvalho, Danilo S. / Valentino, Marco et al. | ArXiv | 2024

    Freier Zugriff

    Modal Decomposition of Hamiltonian Variational Equations

    Wiesel, W. / United States; National Aeronautics and Space Administration / Goddard Space Flight Center | British Library Conference Proceedings | 1994


    Disentangling Latent Factors of Variational Auto-Encoder with Whitening

    Hahn, Sangchul / Choi, Heeyoul | ArXiv | 2018

    Freier Zugriff

    Adaptive Compression of the Latent Space in Variational Autoencoders

    Sejnova, Gabriela / Vavrecka, Michal / Stepanova, Karla | ArXiv | 2023

    Freier Zugriff