The need for large amounts of training and validation data is a huge concern in scaling AI algorithms for autonomous driving. Semantic Image Synthesis (SIS), or label-to-image translation, promises to address this issue by translating semantic layouts to images, providing a controllable generation of photorealistic data. However, they require a large amount of paired data, incurring extra costs. In this work, we present a new task: given a dataset with synthetic images and labels and a dataset with unlabeled real images, our goal is to learn a model that can generate images with the content of the input mask and the appearance of real images. This new task reframes the well-known unsupervised SIS task in a more practical setting, where we leverage cheaply available synthetic data from a driving simulator to learn how to generate photorealistic images of urban scenes. This stands in contrast to previous works, which assume that labels and images come from the same domain but are unpaired during training. We find that previous unsupervised works underperform on this task, as they do not handle distribution shifts between two different domains. To bypass these problems, we propose a novel framework with two main contributions. First, we leverage the synthetic image as a guide to the content of the generated image by penalizing the difference between their high-level features on a patch level. Second, in contrast to previous works which employ one discriminator that overfits the target domain semantic distribution, we employ a discriminator for the whole image and multiscale discriminators on the image patches. Extensive comparisons on the benchmarks benchmarks GTA-V → Cityscapes and GTA-V → Mapillary show the superior performance of the proposed model against state-of-the-art on this task.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Towards Pragmatic Semantic Image Synthesis for Urban Scenes


    Contributors:


    Publication date :

    2023-06-04


    Size :

    7970518 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Real-Time High-Performance Semantic Image Segmentation of Urban Street Scenes

    Dong, Genshun / Yan, Yan / Shen, Chunhua et al. | IEEE | 2021


    CNN based semantic segmentation for urban traffic scenes using fisheye camera

    Deng, Liuyuan / Yang, Ming / Qian, Yeqiang et al. | IEEE | 2017


    A Simple Weight Recall for Semantic Segmentation: Application to Urban Scenes

    LI, Xuhong / DAVOINE, Franck / GRANDVALET, Yves | IEEE | 2018


    Semantic Modeling of Natural Scenes for Content-Based Image Retrieval

    Vogel, J. / Schiele, B. | British Library Online Contents | 2007