Dynamic objects have a significant impact on the robot’s perception of the environment which degrades the performance of essential tasks such as localization and mapping. In this work, we address this problem by synthesizing plausible color, texture and geometry in regions occluded by dynamic objects. We propose the novel geometry-aware DynaFill architecture that follows a coarse-to-fine topology and incorporates our gated reccurrent feedback mechanism to adaptively fuse information from previous timesteps. We optimize our architecture using adversarial training to synthesize fine realistic textures which enables it to hallucinate color and depth structure in occluded regions online in a spatially and temporally coherent manner, without relying on future frame information. Casting our inpainting problem as an image-to-image translation task, our model also corrects regions correlated with the presence of dynamic objects in the scene, such as shadows or reflections. We introduce a large-scale hyperrealistic dataset with RGB-D images, semantic segmentation labels, camera poses as well as groundtruth RGB-D information of occluded regions. Extensive quantitative and qualitative evaluations show that our approach achieves state-of-the-art performance, even in challenging weather conditions. Furthermore, we present results for retrieval-based visual localization with the synthesized images that demonstrate the utility of our approach.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Dynamic Object Removal and Spatio-Temporal RGB-D Inpainting via Geometry-Aware Adversarial Learning


    Contributors:

    Published in:

    Publication date :

    2022-06-01


    Size :

    6943034 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    VeIGAN: Vectorial Inpainting Generative Adversarial Network for Depth Maps Object Removal

    Matias, Lucas P. N. / Sons, Marc / Souza, Jefferson R. et al. | IEEE | 2019


    Spatio-temporal texture synthesis and image inpainting for video applications

    Kumar, S. / Biswas, M. / Belongie, S.J. et al. | IEEE | 2005


    Spatio-Temporal Texture Synthesis and Image Inpainting for Video Applications

    Kumar, S. / Biswas, M. / Belongie, S. J. et al. | British Library Conference Proceedings | 2005


    Object Removal by Exemplar-Based Inpainting

    Criminisi, A. / Perez, P. / Toyama, K. et al. | British Library Conference Proceedings | 2003


    Object removal by exemplar-based inpainting

    Criminisi, A. / Perez, P. / Toyama, K. | IEEE | 2003