Auto-encoding Variational Bayes (AEVB) is a powerful and general algorithm for fitting latent variable models (a promising direction for unsupervised learning), and is well-known for training the Variational Auto-Encoder (VAE). In this tutorial, we focus on motivating AEVB from the classic Expectation Maximization (EM) algorithm, as opposed to from deterministic auto-encoders. Though natural and somewhat self-evident, the connection between EM and AEVB is not emphasized in the recent deep learning literature, and we believe that emphasizing this connection can improve the community's understanding of AEVB. In particular, we find it especially helpful to view (1) optimizing the evidence lower bound (ELBO) with respect to inference parameters as approximate E-step and (2) optimizing ELBO with respect to generative parameters as approximate M-step; doing both simultaneously as in AEVB is then simply tightening and pushing up ELBO at the same time. We discuss how approximate E-step can be interpreted as performing variational inference. Important concepts such as amortization and the reparametrization trick are discussed in great detail. Finally, we derive from scratch the AEVB training procedures of a non-deep and several deep latent variable models, including VAE, Conditional VAE, Gaussian Mixture VAE and Variational RNN. It is our hope that readers would recognize AEVB as a general algorithm that can be used to fit a wide range of latent variable models (not just VAE), and apply AEVB to such models that arise in their own fields of research. PyTorch code for all included models are publicly available.


    Access

    Download


    Export, share and cite



    Title :

    Training Latent Variable Models with Auto-encoding Variational Bayes: A Tutorial


    Contributors:

    Publication date :

    2022


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Joint Training of Variational Auto-Encoder and Latent Energy-Based Model

    Han, Tian / Nijkamp, Erik / Zhou, Linqi et al. | ArXiv | 2020

    Free access

    Disentangling Latent Factors of Variational Auto-Encoder with Whitening

    Hahn, Sangchul / Choi, Heeyoul | ArXiv | 2018

    Free access

    Variational-Bayes Optical Flow

    Chantas, G. | British Library Online Contents | 2014


    Spatial Variational Auto-Encoding via Matrix-Variate Normal Distributions

    Wang, Zhengyang / Yuan, Hao / Ji, Shuiwang | ArXiv | 2017

    Free access

    Estimating Safety by the Empirical Bayes Method: A Tutorial

    Hauer, Ezra / Harwood, Douglas W. / Council, Forrest M. et al. | Transportation Research Record | 2002