Pedestrian Detection is the most critical module of an Autonomous Driving system. Although a camera is commonly used for this purpose, its quality degrades severely in low-light night time driving scenarios. On the other hand, the quality of a thermal camera image remains unaffected in similar conditions. This paper proposes an end-to-end multimodal fusion model for pedestrian detection using RGB and thermal images. Its novel spatio-contextual deep network architecture is capable of exploiting the multimodal input efficiently. It consists of two distinct deformable ResNeXt-50 encoders for feature extraction from the two modalities. Fusion of these two encoded features takes place inside a multimodal feature embedding module (MuFEm) consisting of several groups of a pair of Graph Attention Network and a feature fusion unit. The output of the last feature fusion unit of MuFEm is subsequently passed to two CRFs for their spatial refinement. Further enhancement of the features is achieved by applying channel-wise attention and extraction of contextual information with the help of four RNNs traversing in four different directions. Finally, these feature maps are used by a single-stage decoder to generate the bounding box of each pedestrian and the score map. We have performed extensive experiments of the proposed framework on three publicly available multimodal pedestrian detection benchmark datasets, namely KAIST, CVC-14, and UTokyo. The results on each of them improved the respective state-of-the-art performance. A short video giving an overview of this work along with its qualitative results can be seen at https://youtu.be/FDJdSifuuCs. Our source code will be released upon publication of the paper.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Spatio-Contextual Deep Network-Based Multimodal Pedestrian Detection for Autonomous Driving


    Contributors:


    Publication date :

    2022-09-01


    Size :

    6340926 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English




    The Design of a Pedestrian Aware Contextual Speed Controller for Autonomous Driving

    Jiang, Daniel / Worrall, Stewart / Shan, Mao | IEEE | 2022


    Skeleton-Based Pedestrian Gesture Detection for Autonomous Driving

    Zhang, Mengke / Fei, Wenyu / Yang, Jun | IEEE | 2023


    Pedestrian Detection Using YOLOv5 For Autonomous Driving Applications

    Vikram Reddy, Etikala Raja / Thale, Sushil | IEEE | 2021


    Pedestrian Detection with YOLOv5 in Autonomous Driving Scenario

    Jin, Xianjian / Li, Zhiwei / Yang, Hang | IEEE | 2021