Variational autoencoders (VAEs) are important tools in end-to-end representation learning. VAEs can capture complex data distributions and have been applied extensively in many natural-language-processing (NLP) tasks. However, a common pitfall in sequence-to-sequence learning with VAEs is the posterior-collapse issue in latent space, wherein the model tends to ignore latent variables when a strong auto-regressive decoder is implemented. In this paper, we propose a principled approach to eliminate this issue by applying a discretized bottleneck in the latent space. Specifically, we impose a shared discrete latent space where each input is learned to choose a combination of shared latent atoms as its latent representation. Compared with VAEs employing continuous latent variables, our model endows more promising capability in modeling underlying semantics of discrete sequences and can thus provide more interpretative latent structures. Empirically, we demonstrate the efficiency and effectiveness of our model on a broad range of tasks, including language modeling, unaligned text style transfer, dialog response generation, and neural machine translation.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Discretized Bottleneck in VAE: Posterior-Collapse-Free Sequence-to-Sequence Learning


    Beteiligte:
    Zhao, Yang (Autor:in) / Yu, Ping (Autor:in) / Mahapatra, Suchismit (Autor:in) / Su, Qinliang (Autor:in) / Chen, Changyou (Autor:in)

    Erscheinungsdatum :

    2020


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs

    Long, Teng / Cao, Yanshuai / Cheung, Jackie Chi Kit | ArXiv | 2019

    Freier Zugriff

    Optimum sequence of trains over "bottleneck" section

    Hein, O. | Engineering Index Backfile | 1968


    Bottleneck

    Forschungszentrum Jülich / University of Wuppertal / Cologne University | DataCite | 2009


    Letter-sequence and unit-sequence effects during learning and retention.

    Mc Cullough, T. A. / Turnage, T. W. | NTRS | 1968


    When Is a Bottleneck a Bottleneck?

    Schadschneider, Andreas / Schmidt, Johannes / Popkov, Vladislav | Springer Verlag | 2016