Deep generative neural networks, such as Variational AutoEncoders (VAEs), offer an opportunity to better understand and control language models from the perspective of sentence-level latent spaces. To combine the controllability of VAE latent spaces with the state-of-the-art performance of recent large language models (LLMs), we present in this work LlaMaVAE, which combines expressive encoder and decoder models (sentenceT5 and LlaMA) with a VAE architecture, aiming to provide better text generation control to LLMs. In addition, to conditionally guide the VAE generation, we investigate a new approach based on flow-based invertible neural networks (INNs) named Invertible CVAE. Experimental results reveal that LlaMaVAE can outperform the previous state-of-the-art VAE language model, Optimus, across various tasks, including language modelling, semantic textual similarity and definition modelling. Qualitative analysis on interpolation and traversal experiments also indicates an increased degree of semantic clustering and geometric consistency, which enables better generation control.


    Access

    Download


    Export, share and cite



    Title :

    LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spaces


    Contributors:

    Publication date :

    2023


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    SENTENCE GENERATION DEVICE, SENTENCE GENERATION METHOD, AND RECORDING MEDIUM

    UMANO HIROMICHI / EGAWA CHISATO | European Patent Office | 2024

    Free access

    GUIDANCE SENTENCE GENERATION DEVICE, GUIDE SYSTEM, AND GUIDANCE SENTENCE GENERATION METHOD AND PROGRAM

    MIYAKE ASUKA / HIRAO MISA / WATANABE MASAHIRO et al. | European Patent Office | 2019

    Free access

    Large-scale parking lot vehicle guiding system and guiding method thereof

    ZHANG ZHENYU / ZHENG LINXIN / LI QIANG | European Patent Office | 2016

    Free access

    Cross-modal Variational Alignment of Latent Spaces

    Theodoridis, Thomas / Chatzis, Theocharis / Solachidis, Vassilios et al. | IEEE | 2020