We study outdoor 3D scene understanding, a challenging task demanding the intelligent system to infer both geometry and semantics from a single-view image – a critical skill for autonomous vehicles to navigate in the real 3D world. Towards this end, we present an instance-aware monocular semantic scene completion framework. To the best of our knowledge, this is the first endeavor specifically targeting the challenge of instance perception in the camera-based semantic scene completion task. Our method consists of two stages. In stage I, we design a region-based VQ-VAE network, providing an effective solution for 3D occupancy prediction. In stage II, we first introduce an instance-aware attention module, explicitly incorporating instance-level cues captured from mask images to enhance the instance features in RGB images. Then we leverage the deformable cross-attention to aggregate image features corresponding to each voxel query and utilize the deformable self-attention to refine query proposals. We combine these key ingredients and evaluate our method on two challenging datasets, namely SemanticKITTI and SSCBench-KITTI-360. The results unequivocally demonstrate the superiority of our proposed method over the state-of-the-art VoxFormer-S. Specifically, our method surpasses VoxFormer-S by 0.22 IoU and 0.72 mIoU on the validation set and achieves an impressive improvement of 3.04 IoU and 1.06 mIoU on the SSCBench-KITTI-360 validation set. Meanwhile, our approach ensures accurate perception of critical instances, thereby exhibiting its exceptional performance and potential for practical deployment.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Instance-Aware Monocular 3D Semantic Scene Completion


    Beteiligte:
    Xiao, Haihong (Autor:in) / Xu, Hongbin (Autor:in) / Kang, Wenxiong (Autor:in) / Li, Yuqiong (Autor:in)


    Erscheinungsdatum :

    2024-07-01


    Format / Umfang :

    3066980 byte




    Medientyp :

    Aufsatz (Zeitschrift)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    SPARSE VOXEL TRANSFORMER FOR CAMERA-BASED 3D SEMANTIC SCENE COMPLETION

    LI YIMING / YU ZHIDING / CHOY CHRISTOPHER B et al. | Europäisches Patentamt | 2024

    Freier Zugriff

    Instance-Aware Semantic Segmentation of Road Furniture in Mobile Laser Scanning Data

    Li, Fashuai / Zhou, Zhize / Xiao, Jianhua et al. | IEEE | 2022



    Incorporating scene priors to dense monocular mapping

    Concha, A. | British Library Online Contents | 2015


    Semantic segmentation of infrared ships based on scene-aware priors

    Hu, Junwei / Zhang, Feng / Zhang, Jinsuo et al. | British Library Conference Proceedings | 2023