The Variational Autoencoder (VAE) is a popular and powerful model applied to text modelling to generate diverse sentences. However, an issue known as posterior collapse (or KL loss vanishing) happens when the VAE is used in text modelling, where the approximate posterior collapses to the prior, and the model will totally ignore the latent variables and be degraded to a plain language model during text generation. Such an issue is particularly prevalent when RNN-based VAE models are employed for text modelling. In this paper, we propose a simple, generic architecture called Timestep-Wise Regularisation VAE (TWR-VAE), which can effectively avoid posterior collapse and can be applied to any RNN-based VAE models. The effectiveness and versatility of our model are demonstrated in different tasks, including language modelling and dialogue response generation.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation


    Beteiligte:
    Li, Ruizhe (Autor:in) / Li, Xiao (Autor:in) / Chen, Guanyi (Autor:in) / Lin, Chenghua (Autor:in)

    Erscheinungsdatum :

    2020


    Anmerkungen:

    Accepted by COLING 2020, final camera ready version


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Timestep Control for Weakly Instationary Flows

    Steiner, Christina / Noelle, Sebastian | Springer Verlag | 2010


    Variational Autoencoder

    Pinheiro Cinelli, Lucas / Araújo Marins, Matheus / Barros da Silva, Eduardo Antúnio et al. | Springer Verlag | 2021


    Variational Autoencoder

    Okadome, Takeshi | Springer Verlag | 2025


    Autoencoding Variational Autoencoder

    Cemgil, A. Taylan / Ghaisas, Sumedh / Dvijotham, Krishnamurthy et al. | ArXiv | 2020

    Freier Zugriff

    Variational Lossy Autoencoder

    Chen, Xi / Kingma, Diederik P. / Salimans, Tim et al. | ArXiv | 2016

    Freier Zugriff