Video reconstruction refers to generate videos through the high-level representations (edge map, labels and so on), while the reconstruction quality is always unsatisfactory due to sparse high-level representations, especially on video data. In order to improve the video reconstruction quality, we proposed a novel approach that generates realistic video from its multimodal information including structure features and color features. To extract color features, we mainly apply the k-means algorithm to segment labels and the structure features are extracted by an edge detection network. Video generation is regarded as learning the mapping from multimodal representations to the original videos. So, a conditional GAN is applied with a learning objective that models the temporal video dynamics. We use a spatio-temporal generator with attention to model the inter-frame dynamics and video consistency is improved in this way. Moreover, we use a multiscale discriminator to improve the improve the intra-frame quality of the video. Experimental results on Cityscapes, Apolloscape datasets demonstrate that our proposed approach performs better in both traditional and generative evaluating indicators.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Video Reconstruction with Multimodal Information


    Contributors:
    Xie, Zhipeng (author) / Duan, Yiping (author) / Du, Qiyuan (author) / Tao, Xiaoming (author) / Yu, Jiazhong (author)


    Publication date :

    2023-10-10


    Size :

    2991586 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Hierarchical & multimodal video captioning: Discovering and transferring multimodal knowledge for vision to language

    Liu, An-An / Xu, Ning / Wong, Yongkang et al. | British Library Online Contents | 2017


    Optimal Reconstruction of Human Motion From Scarce Multimodal Data

    Giuseppe Averta / Matilde Iuculano / Paolo Salaris et al. | BASE | 2022

    Free access

    Information enquiry kiosk with multimodal user interface

    Karpov, A. A. / Ronzhin, A. L. | British Library Online Contents | 2009



    Construction of Multimodal Transport Information Platform

    Wang Ya / Cheng Yu / Zhao Zhi | DOAJ | 2018

    Free access