In variational autoencoders (VAEs), the variational posterior often collapses to the prior, known as posterior collapse, which leads to poor representation learning quality. An adjustable hyperparameter beta has been introduced in VAEs to address this issue. This study sharply evaluates the conditions under which the posterior collapse occurs with respect to beta and dataset size by analyzing a minimal VAE in a high-dimensional limit. Additionally, this setting enables the evaluation of the rate-distortion curve of the VAE. Our results show that, unlike typical regularization parameters, VAEs face "inevitable posterior collapse" beyond a certain beta threshold, regardless of dataset size. Moreover, the dataset-size dependence of the derived rate-distortion curve suggests that relatively large datasets are required to achieve a rate-distortion curve with high rates. These findings robustly explain generalization behavior observed in various real datasets with highly non-linear VAEs.


    Access

    Download


    Export, share and cite



    Title :

    High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve


    Contributors:

    Publication date :

    2023


    Remarks:

    25 pages, 7 figures


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs

    Long, Teng / Cao, Yanshuai / Cheung, Jackie Chi Kit | ArXiv | 2019

    Free access

    Mitigating Modality Collapse in Multimodal VAEs Via Impartial Optimization

    Javaloy, Adrian / Meghdadi, Maryam / Valera, Isabel | TIBKAT | 2023


    f-VAEs: Improve VAEs with Conditional Flows

    Su, Jianlin / Wu, Guang | ArXiv | 2018

    Free access

    Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization

    Javaloy, Adrián / Meghdadi, Maryam / Valera, Isabel | ArXiv | 2022

    Free access

    Beyond Diagonal Covariance: Flexible Posterior VAEs via Free-Form Injective Flows

    Sorrenson, Peter / Lührs, Lukas / Olischläger, Hans et al. | ArXiv | 2025

    Free access