In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first network is a multi modal variational autoencoder that maps directly one modality to the other, while the second one is a single-modal variational autoencoder. In order to associate the two spaces, we apply variational alignment, which acts as a translation mechanism that projects the latent space of the first VAE onto the one of the single-modal VAE through an intermediate distribution. Experimental results on four well-known datasets, covering two different application domains (food image analysis and 3D hand pose estimation), show the generality of the proposed method and its superiority against a number of state-of-the-art approaches.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Cross-modal Variational Alignment of Latent Spaces




    Publication date :

    2020-06-01


    Size :

    1009563 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Cross-Modal Generation and Pair Correlation Alignment Hashing

    Ou, Weihua / Deng, Jiaxin / Zhang, Lei et al. | IEEE | 2023


    Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders

    Zhang, Yingji / Carvalho, Danilo S. / Valentino, Marco et al. | ArXiv | 2024

    Free access

    Modal Decomposition of Hamiltonian Variational Equations

    Wiesel, W. / United States; National Aeronautics and Space Administration / Goddard Space Flight Center | British Library Conference Proceedings | 1994


    Adaptive Compression of the Latent Space in Variational Autoencoders

    Sejnova, Gabriela / Vavrecka, Michal / Stepanova, Karla | ArXiv | 2023

    Free access

    Disentangling Latent Factors of Variational Auto-Encoder with Whitening

    Hahn, Sangchul / Choi, Heeyoul | ArXiv | 2018

    Free access