Forecasting or estimating the future scenes is an important task in autonomous driving. The future scene is predicted using the current and past scenes. Typically, the scene is forecast as either visible camera images or semantic maps. The forecast scene can be used for planning, navigation and control tasks. However, the scene forecast in the form of visible images are susceptible to varying illumination, varying environmental and adverse weather conditions. In our work, we address this limitation using a novel deep learning based visible and thermal camera forecasting algorithm termed as the Seq2Seq. The Seq2Seq, a conditional GAN framework, forecasts the visible as well as the thermal images using the current and past visible-thermal camera images. The generator model is a deep sensor fusion based on the encoder-decoder architecture with a convolution LSTM branch. The discriminator model is also a deep sensor fusion based on the patchGAN architecture. The Seq2Seq is validated using the KAIST public dataset. The results show that the proposed framework can accurately forecast the future visible and thermal images. Moreover, we also demonstrate the application of the Seq2Seq, by performing semantic segmentation on the forecast visible-thermal images using the MFNet.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Deep Fusion-based Visible and Thermal Camera Forecasting using Seq2Seq GAN


    Contributors:


    Publication date :

    2021-07-11


    Size :

    6108614 byte




    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    COMBINING VISIBLE LIGHT CAMERA AND THERMAL CAMERA INFORMATION

    KUNDU KUMAR SUBRATA / BANGALORE RAMAIAH KUMAR NAVEEN | European Patent Office | 2022

    Free access

    THE MEASUREMENT METHOD OF METRO RAIL CORRUGATION BASED ON THE SEQ2SEQ MODEL AND THE FUSION DATA OF VIBRATION AND NOISE

    XIAO HONG / WANG YANG / GAO LIANG et al. | European Patent Office | 2024

    Free access

    A Novel Long-time Prediction Car-following Model Based Mixed Seq2seq Architecture

    Fang, Shan / Yang, Lan / Zhao, Xiangmo et al. | IEEE | 2022


    A Real-Time Seq2Seq Beamforming Prediction Model for C-V2X Links

    Xiang, Weidong / Elangovan, Vivekanandh / Lakshmanan, Sridhar | Springer Verlag | 2022


    Deep fusion of camera and LIDAR

    Schneider, Nick | TIBKAT | 2023

    Free access