The Variational Autoencoder (VAE) is a popular and powerful model applied to text modelling to generate diverse sentences. However, an issue known as posterior collapse (or KL loss vanishing) happens when the VAE is used in text modelling, where the approximate posterior collapses to the prior, and the model will totally ignore the latent variables and be degraded to a plain language model during text generation. Such an issue is particularly prevalent when RNN-based VAE models are employed for text modelling. In this paper, we propose a simple, generic architecture called Timestep-Wise Regularisation VAE (TWR-VAE), which can effectively avoid posterior collapse and can be applied to any RNN-based VAE models. The effectiveness and versatility of our model are demonstrated in different tasks, including language modelling and dialogue response generation.


    Access

    Download


    Export, share and cite



    Title :

    Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation


    Contributors:
    Li, Ruizhe (author) / Li, Xiao (author) / Chen, Guanyi (author) / Lin, Chenghua (author)

    Publication date :

    2020


    Remarks:

    Accepted by COLING 2020, final camera ready version


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Timestep Control for Weakly Instationary Flows

    Steiner, Christina / Noelle, Sebastian | Springer Verlag | 2010


    Variational Autoencoder

    Okadome, Takeshi | Springer Verlag | 2025


    Variational Autoencoder

    Pinheiro Cinelli, Lucas / Araújo Marins, Matheus / Barros da Silva, Eduardo Antúnio et al. | Springer Verlag | 2021


    Autoencoding Variational Autoencoder

    Cemgil, A. Taylan / Ghaisas, Sumedh / Dvijotham, Krishnamurthy et al. | ArXiv | 2020

    Free access

    Variational Lossy Autoencoder

    Chen, Xi / Kingma, Diederik P. / Salimans, Tim et al. | ArXiv | 2016

    Free access