Variational Autoencoders (VAEs) are a popular framework for unsupervised learning and data generation. A plethora of methods have been proposed focusing on improving VAEs, with the incorporation of adversarial objectives and the integration of prior learning mechanisms being prominent directions. When it comes to the former, an indicative instance is the recently introduced family of Introspective VAEs aiming at ensuring that a low likelihood is assigned to unrealistic samples. In this study, we focus on the Soft-IntroVAE (S-IntroVAE), one of only two members of the Introspective VAE family, the other being the original IntroVAE. We select S-IntroVAE for its state-of-the-art status and its training stability. In particular, we investigate the implication of incorporating a multimodal and trainable prior into this S-IntroVAE. Namely, we formulate the prior as a third player and show that when trained in cooperation with the decoder constitutes an effective way for prior learning, which shares the Nash Equilibrium with the vanilla S-IntroVAE. Furthermore, based on a modified formulation of the optimal ELBO in S-IntroVAE, we develop theoretically motivated regularizations, namely (i) adaptive variance clipping to stabilize training when learning the prior and (ii) responsibility regularization to discourage the formation of inactive prior modes. Finally, we perform a series of targeted experiments on a 2D density estimation benchmark and in an image generation setting comprised of the (F)-MNIST and CIFAR-10 datasets demonstrating the effect of prior learning in S-IntroVAE in generation and representation learning.


    Access

    Download


    Export, share and cite



    Title :

    Prior Learning in Introspective VAEs


    Contributors:

    Publication date :

    2024


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    LEARNING HIERARCHICAL PRIORS IN VAES

    Klushyn, Alexej / Chen, Nutan / Kurle, Richard et al. | TIBKAT | 2020


    f-VAEs: Improve VAEs with Conditional Flows

    Su, Jianlin / Wu, Guang | ArXiv | 2018

    Free access

    Relaxed-Responsibility Hierarchical Discrete VAEs

    Willetts, Matthew / Miscouridou, Xenia / Roberts, Stephen et al. | ArXiv | 2020

    Free access

    LEARNING FLAT LATENT MANIFOLDS WITH VAES

    Chen, Nutan / Klushyn, Alexej / Ferroni, Francesco et al. | TIBKAT | 2021


    On Incorporating Inductive Biases into VAEs

    Miao, Ning / Mathieu, Emile / Siddharth, N. et al. | ArXiv | 2021

    Free access