Driving simulators play a large role in developing and testing new intelligent vehicle systems. The visual fidelity of the simulation is critical for building vision-based algorithms and conducting human driver experiments. Low visual fidelity breaks immersion for human-in-the-loop driving experiments. Conventional computer graphics pipelines use detailed 3D models, meshes, textures, and rendering engines to generate 2D images from 3D scenes. These processes are labor-intensive, and they do not generate photorealistic imagery. Here we introduce a hybrid generative neural graphics pipeline for improving the visual fidelity of driving simulations. Given a 3D scene, we partially-render only important objects of interest, such as vehicles, and use generative adversarial processes to synthesize the background and the rest of the image. To this end, we propose a novel image formation strategy to form 2D semantic images from 3D scenery consisting of simple object models without textures. These semantic images are then converted into photorealistic RGB images with a state-of-the-art Generative Adversarial Network (GAN) trained on real-world driving scenes. This replaces repetitiveness with randomly generated but photorealistic surfaces. Finally, the partially-rendered and GAN synthesized images are blended with a blending GAN. We show that the photorealism of images generated with the proposed method is more similar to real-world driving datasets such as Cityscapes and KITTI than conventional approaches. This comparison is made using semantic retention analysis and Frechet Inception Distance (FID) measurements.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Photorealism in Driving Simulations: Blending Generative Adversarial Image Synthesis With Rendering


    Contributors:

    Published in:

    Publication date :

    2022-12-01


    Size :

    6604241 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English




    Generative adversarial network enriched driving simulation

    SONG HAO / PENG JUN / DENG NENGXIU et al. | European Patent Office | 2022

    Free access

    GENERATIVE ADVERSARIAL NETWORK ENRICHED DRIVING SIMULATION

    SONG HAO / PENG JUN / DENG NENGXIU et al. | European Patent Office | 2020

    Free access

    GENERATIVE ADVERSARIAL NETWORK ENRICHED DRIVING SIMULATION

    SONG HAO / PENG JUN / DENG NENGXIU et al. | European Patent Office | 2020

    Free access

    GENERATIVE ADVERSARIAL NETWORK ENRICHED DRIVING SIMULATION

    SONG HAO / PENG JUN / DENG NENGXIU et al. | European Patent Office | 2022

    Free access