Learning a robust video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation. Directly applying image VAEs to individual frames in isolation can result in temporal inconsistencies and suboptimal compression rates due to a lack of temporal compression. Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance. In this paper, we present a novel and powerful video autoencoder capable of high-fidelity video encoding. First, we observe that entangling spatial and temporal compression by merely extending the image VAE to a 3D VAE can introduce motion blur and detail distortion artifacts. Thus, we propose temporal-aware spatial compression to better encode and decode the spatial information. Additionally, we integrate a lightweight motion compression model for further temporal compression. Second, we propose to leverage the textual information inherent in text-to-video datasets and incorporate text guidance into our model. This significantly enhances reconstruction quality, particularly in terms of detail preservation and temporal stability. Third, we further improve the versatility of our model through joint training on both images and videos, which not only enhances reconstruction quality but also enables the model to perform both image and video autoencoding. Extensive evaluations against strong recent baselines demonstrate the superior performance of our method. The project website can be found at~\href{https://yzxing87.github.io/vae/}{https://yzxing87.github.io/vae/}.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Large Motion Video Autoencoding with Cross-modal Video VAE


    Beteiligte:
    Xing, Yazhou (Autor:in) / Fei, Yang (Autor:in) / He, Yingqing (Autor:in) / Chen, Jingye (Autor:in) / Xie, Jiaxin (Autor:in) / Chi, Xiaowei (Autor:in) / Chen, Qifeng (Autor:in)

    Erscheinungsdatum :

    2024


    Anmerkungen:

    Project Website: https://yzxing87.github.io/vae/


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Autoencoding Variational Autoencoder

    Cemgil, A. Taylan / Ghaisas, Sumedh / Dvijotham, Krishnamurthy et al. | ArXiv | 2020

    Freier Zugriff

    AUTOENCODING FEATURES FOR AVIATION MACHINE LEARNING PROBLEMS

    Wang, Liya / Lucic, Panta / Campbell, Keith et al. | TIBKAT | 2021


    Autoencoding Features for Aviation Machine Learning Problems

    Wang, Liya / Lucic, Panta / Campbell, Keith et al. | AIAA | 2021


    Autoencoding Graph Neural Networks for Scalable Transceiver Design

    Kim, Junbeom / Lee, Hoon / Park, Seok-Hwan | IEEE | 2022


    To the Moon! Space Launch System Modal Testing With Video and Motion Magnification

    Justin G. Chen / Raul Rios / Kevin E. Franks et al. | NTRS