Understanding urban scenes is a fundamental ability requirement for assisted driving and autonomous vehicles. Most of the available urban scene understanding methods use red-green-blue (RGB) images; however, their segmentation performances are prone to degradation under adverse lighting conditions. Recently, many effective artificial neural networks have been presented for urban scene understanding and have shown that incorporating RGB and thermal (RGB-T) images can improve segmentation accuracy even under unsatisfactory lighting conditions. However, the potential of multimodal feature fusion has not been fully exploited because operations such as simply concatenating the RGB and thermal features or averaging their maps have been adopted. To improve the fusion of multimodal features and the segmentation accuracy, we propose a multitask-aware network (MTANet) with hierarchical multimodal fusion (multiscale fusion strategy) for RGB-T urban scene understanding. We developed a hierarchical multimodal fusion module to enhance feature fusion and built a high-level semantic module to extract semantic information for merging with coarse features at various abstraction levels. Using the multilevel fusion module, we exploited low-, mid-, and high-level fusion to improve segmentation accuracy. The multitask module uses boundary, binary, and semantic supervision to optimize the MTANet parameters. Extensive experiments were performed on two benchmark RGB-T datasets to verify the improved performance of the proposed MTANet compared with state-of-the-art methods.1


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    MTANet: Multitask-Aware Network With Hierarchical Multimodal Fusion for RGB-T Urban Scene Understanding


    Contributors:
    Zhou, Wujie (author) / Dong, Shaohua (author) / Lei, Jingsheng (author) / Yu, Lu (author)

    Published in:

    Publication date :

    2023-01-01


    Size :

    4193183 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Multimodal information fusion for urban scene understanding

    Xu, P. | British Library Online Contents | 2016


    EGFNet: Edge-Aware Guidance Fusion Network for RGB–Thermal Urban Scene Parsing

    Dong, Shaohua / Zhou, Wujie / Xu, Caie et al. | IEEE | 2024



    Hierarchical Scene Understanding for Intelligent Vehicles

    Spehr, J. / Rosebrock, D. / Wahl, F.M. et al. | British Library Conference Proceedings | 2011


    Hierarchical Bayesian models for unsupervised scene understanding

    Steinberg, D. M. / Pizarro, O. / Williams, S. B. | British Library Online Contents | 2015