We study outdoor 3D scene understanding, a challenging task demanding the intelligent system to infer both geometry and semantics from a single-view image – a critical skill for autonomous vehicles to navigate in the real 3D world. Towards this end, we present an instance-aware monocular semantic scene completion framework. To the best of our knowledge, this is the first endeavor specifically targeting the challenge of instance perception in the camera-based semantic scene completion task. Our method consists of two stages. In stage I, we design a region-based VQ-VAE network, providing an effective solution for 3D occupancy prediction. In stage II, we first introduce an instance-aware attention module, explicitly incorporating instance-level cues captured from mask images to enhance the instance features in RGB images. Then we leverage the deformable cross-attention to aggregate image features corresponding to each voxel query and utilize the deformable self-attention to refine query proposals. We combine these key ingredients and evaluate our method on two challenging datasets, namely SemanticKITTI and SSCBench-KITTI-360. The results unequivocally demonstrate the superiority of our proposed method over the state-of-the-art VoxFormer-S. Specifically, our method surpasses VoxFormer-S by 0.22 IoU and 0.72 mIoU on the validation set and achieves an impressive improvement of 3.04 IoU and 1.06 mIoU on the SSCBench-KITTI-360 validation set. Meanwhile, our approach ensures accurate perception of critical instances, thereby exhibiting its exceptional performance and potential for practical deployment.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Instance-Aware Monocular 3D Semantic Scene Completion


    Contributors:
    Xiao, Haihong (author) / Xu, Hongbin (author) / Kang, Wenxiong (author) / Li, Yuqiong (author)


    Publication date :

    2024-07-01


    Size :

    3066980 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    SPARSE VOXEL TRANSFORMER FOR CAMERA-BASED 3D SEMANTIC SCENE COMPLETION

    LI YIMING / YU ZHIDING / CHOY CHRISTOPHER B et al. | European Patent Office | 2024

    Free access

    Instance-Aware Semantic Segmentation of Road Furniture in Mobile Laser Scanning Data

    Li, Fashuai / Zhou, Zhize / Xiao, Jianhua et al. | IEEE | 2022



    Incorporating scene priors to dense monocular mapping

    Concha, A. | British Library Online Contents | 2015


    Semantic segmentation of infrared ships based on scene-aware priors

    Hu, Junwei / Zhang, Feng / Zhang, Jinsuo et al. | British Library Conference Proceedings | 2023