Forecasting or estimating the future scenes is an important task in autonomous driving. The future scene is predicted using the current and past scenes. Typically, the scene is forecast as either visible camera images or semantic maps. The forecast scene can be used for planning, navigation and control tasks. However, the scene forecast in the form of visible images are susceptible to varying illumination, varying environmental and adverse weather conditions. In our work, we address this limitation using a novel deep learning based visible and thermal camera forecasting algorithm termed as the Seq2Seq. The Seq2Seq, a conditional GAN framework, forecasts the visible as well as the thermal images using the current and past visible-thermal camera images. The generator model is a deep sensor fusion based on the encoder-decoder architecture with a convolution LSTM branch. The discriminator model is also a deep sensor fusion based on the patchGAN architecture. The Seq2Seq is validated using the KAIST public dataset. The results show that the proposed framework can accurately forecast the future visible and thermal images. Moreover, we also demonstrate the application of the Seq2Seq, by performing semantic segmentation on the forecast visible-thermal images using the MFNet.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Deep Fusion-based Visible and Thermal Camera Forecasting using Seq2Seq GAN


    Beteiligte:
    John, Vijay (Autor:in) / Lakshmanan, Annamalai (Autor:in) / Boyali, Ali (Autor:in) / Thompson, Simon (Autor:in) / Mita, Seiichi (Autor:in)


    Erscheinungsdatum :

    11.07.2021


    Format / Umfang :

    6108614 byte




    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    COMBINING VISIBLE LIGHT CAMERA AND THERMAL CAMERA INFORMATION

    KUNDU KUMAR SUBRATA / BANGALORE RAMAIAH KUMAR NAVEEN | Europäisches Patentamt | 2022

    Freier Zugriff

    THE MEASUREMENT METHOD OF METRO RAIL CORRUGATION BASED ON THE SEQ2SEQ MODEL AND THE FUSION DATA OF VIBRATION AND NOISE

    XIAO HONG / WANG YANG / GAO LIANG et al. | Europäisches Patentamt | 2024

    Freier Zugriff

    A Novel Long-time Prediction Car-following Model Based Mixed Seq2seq Architecture

    Fang, Shan / Yang, Lan / Zhao, Xiangmo et al. | IEEE | 2022


    A Real-Time Seq2Seq Beamforming Prediction Model for C-V2X Links

    Xiang, Weidong / Elangovan, Vivekanandh / Lakshmanan, Sridhar | Springer Verlag | 2022


    Deep fusion of camera and LIDAR

    Schneider, Nick | TIBKAT | 2023

    Freier Zugriff