We explain why directly changing the prior can be a surprisingly ineffective mechanism for incorporating inductive biases into VAEs, and introduce a simple and effective alternative approach: Intermediary Latent Space VAEs(InteL-VAEs). InteL-VAEs use an intermediary set of latent variables to control the stochasticity of the encoding process, before mapping these in turn to the latent representation using a parametric function that encapsulates our desired inductive bias(es). This allows us to impose properties like sparsity or clustering on learned representations, and incorporate human knowledge into the generative model. Whereas changing the prior only indirectly encourages behavior through regularizing the encoder, InteL-VAEs are able to directly enforce desired characteristics. Moreover, they bypass the computation and encoder design issues caused by non-Gaussian priors, while allowing for additional flexibility through training of the parametric mapping function. We show that these advantages, in turn, lead to both better generative models and better representations being learned.


    Access

    Download


    Export, share and cite



    Title :

    On Incorporating Inductive Biases into VAEs


    Contributors:

    Publication date :

    2021


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    f-VAEs: Improve VAEs with Conditional Flows

    Su, Jianlin / Wu, Guang | ArXiv | 2018

    Free access

    Disentangling Granularity: An Implicit Inductive Bias in Factorized VAEs

    Chen, Zihao / Xiang, Yu / Wang, Wenyong | ArXiv | 2025

    Free access

    LEARNING HIERARCHICAL PRIORS IN VAES

    Klushyn, Alexej / Chen, Nutan / Kurle, Richard et al. | TIBKAT | 2020


    Relaxed-Responsibility Hierarchical Discrete VAEs

    Willetts, Matthew / Miscouridou, Xenia / Roberts, Stephen et al. | ArXiv | 2020

    Free access

    Prior Learning in Introspective VAEs

    Athanasiadis, Ioannis / Lindsten, Fredrik / Felsberg, Michael | ArXiv | 2024

    Free access