The posterior collapse phenomenon in variational autoencoder (VAE), where the variational posterior distribution closely matches the prior distribution, can hinder the quality of the learned latent variables. As a consequence of posterior collapse, the latent variables extracted by the encoder in VAE preserve less information from the input data and thus fail to produce meaningful representations as input to the reconstruction process in the decoder. While this phenomenon has been an actively addressed topic related to VAE performance, the theory for posterior collapse remains underdeveloped, especially beyond the standard VAE. In this work, we advance the theoretical understanding of posterior collapse to two important and prevalent yet less studied classes of VAE: conditional VAE and hierarchical VAE. Specifically, via a non-trivial theoretical analysis of linear conditional VAE and hierarchical VAE with two levels of latent, we prove that the cause of posterior collapses in these models includes the correlation between the input and output of the conditional VAE and the effect of learnable encoder variance in the hierarchical VAE. We empirically validate our theoretical findings for linear conditional and hierarchical VAE and demonstrate that these results are also predictive for non-linear cases with extensive experiments.


    Access

    Download


    Export, share and cite



    Title :

    Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders


    Contributors:
    Dang, Hien (author) / Tran, Tho (author) / Nguyen, Tan (author) / Ho, Nhat (author)

    Publication date :

    2023


    Remarks:

    Accepted (Poster) at the Twelfth International Conference on Learning Representations


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Variational Autoencoders

    Ghojogh, Benyamin / Crowley, Mark / Karray, Fakhri et al. | Springer Verlag | 2022


    Learning Conditional Variational Autoencoders with Missing Covariates

    Ramchandran, Siddharth / Tikhonov, Gleb / Lönnroth, Otto et al. | ArXiv | 2022

    Free access

    Deep Tracking Portfolios Using Autoencoders and Variational Autoencoders

    Urrego, Daniel Aragón / Nieto, Oscar Eduardo Reyes / Quimbayo, Carlos Andrés Zapata | Springer Verlag | 2024


    Certifiably Robust Variational Autoencoders

    Barrett, Ben / Camuto, Alexander / Willetts, Matthew et al. | ArXiv | 2021

    Free access

    Mixed-curvature Variational Autoencoders

    Skopek, Ondrej / Ganea, Octavian-Eugen / Bécigneul, Gary | ArXiv | 2019

    Free access