Understanding urban scenes is a fundamental ability requirement for assisted driving and autonomous vehicles. Most of the available urban scene understanding methods use red-green-blue (RGB) images; however, their segmentation performances are prone to degradation under adverse lighting conditions. Recently, many effective artificial neural networks have been presented for urban scene understanding and have shown that incorporating RGB and thermal (RGB-T) images can improve segmentation accuracy even under unsatisfactory lighting conditions. However, the potential of multimodal feature fusion has not been fully exploited because operations such as simply concatenating the RGB and thermal features or averaging their maps have been adopted. To improve the fusion of multimodal features and the segmentation accuracy, we propose a multitask-aware network (MTANet) with hierarchical multimodal fusion (multiscale fusion strategy) for RGB-T urban scene understanding. We developed a hierarchical multimodal fusion module to enhance feature fusion and built a high-level semantic module to extract semantic information for merging with coarse features at various abstraction levels. Using the multilevel fusion module, we exploited low-, mid-, and high-level fusion to improve segmentation accuracy. The multitask module uses boundary, binary, and semantic supervision to optimize the MTANet parameters. Extensive experiments were performed on two benchmark RGB-T datasets to verify the improved performance of the proposed MTANet compared with state-of-the-art methods.1
MTANet: Multitask-Aware Network With Hierarchical Multimodal Fusion for RGB-T Urban Scene Understanding
IEEE Transactions on Intelligent Vehicles ; 8 , 1 ; 48-58
2023-01-01
4193183 byte
Aufsatz (Zeitschrift)
Elektronische Ressource
Englisch
Multimodal information fusion for urban scene understanding
British Library Online Contents | 2016
|Hierarchical Scene Understanding for Intelligent Vehicles
British Library Conference Proceedings | 2011
|Hierarchical Bayesian models for unsupervised scene understanding
British Library Online Contents | 2015
|