A number of variational autoencoders (VAEs) have recently emerged with the aim of modeling multimodal data, e.g., to jointly model images and their corresponding captions. Still, multimodal VAEs tend to focus solely on a subset of the modalities, e.g., by fitting the image while neglecting the caption. We refer to this limitation as modality collapse. In this work, we argue that this effect is a consequence of conflicting gradients during multimodal VAE training. We show how to detect the sub-graphs in the computational graphs where gradients conflict (impartiality blocks), as well as how to leverage existing gradient-conflict solutions from multitask learning to mitigate modality collapse. That is, to ensure impartial optimization across modalities. We apply our training framework to several multimodal VAE models, losses and datasets from the literature, and empirically show that our framework significantly improves the reconstruction performance, conditional generation, and coherence of the latent space across modalities.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization


    Beteiligte:
    Javaloy, Adrián (Autor:in) / Meghdadi, Maryam (Autor:in) / Valera, Isabel (Autor:in)

    Erscheinungsdatum :

    2022


    Anmerkungen:

    Accepted as a Spotlight paper at ICML 2022. 27 pages, 10 figures


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Mitigating Modality Collapse in Multimodal VAEs Via Impartial Optimization

    Javaloy, Adrian / Meghdadi, Maryam / Valera, Isabel | TIBKAT | 2023


    On the Limitations of Multimodal VAEs

    Daunhawer, Imant / Sutter, Thomas M. / Chin-Cheong, Kieran et al. | ArXiv | 2021

    Freier Zugriff

    f-VAEs: Improve VAEs with Conditional Flows

    Su, Jianlin / Wu, Guang | ArXiv | 2018

    Freier Zugriff

    On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs

    Long, Teng / Cao, Yanshuai / Cheung, Jackie Chi Kit | ArXiv | 2019

    Freier Zugriff

    LEARNING HIERARCHICAL PRIORS IN VAES

    Klushyn, Alexej / Chen, Nutan / Kurle, Richard et al. | TIBKAT | 2020