Depth-aware panoptic segmentation reconstructs the 3D scene from a single image through the tasks of monocular depth estimation and panoptic segmentation. Recent works have designed unified networks where both tasks are formulated as per-pixel prediction. Depth is usually learned via regression while segmentation is posed as per-pixel classification. However, panoptic segmentation benchmarks have been recently dominated by methods that employ an alternative formulation and solve instance and semantic segmentation in a unified manner with mask classification. Motivated by this insight and by the recent success of hybrid classification-regression depth estimation methods based on adaptive bins, we propose a novel unified network based on Mask Transformer for depth-aware panoptic segmentation. Instead of predicting a global depth distribution per image, we leverage cross-task information sharing and equip the network with novel panoptic bins which perform depth discretization at panoptic mask-level. Consequently, the network can better understand the 3D structure of the scene at a more detailed level, leading to depth predictions that are more aligned with the visible objects. Moreover, image embeddings are efficiently shared by both tasks, leading to synergy during training and improved performance. We perform extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets and demonstrate that the proposed Mask Transformer network with panoptic adaptive bins achieves state-of-the-art results and more accurate predictions than per-pixel dense prediction methods.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Depth-Aware Panoptic Segmentation with Mask Transformers and Panoptic Bins for Autonomous Driving


    Contributors:


    Publication date :

    2024-09-24


    Size :

    4108730 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Location-Guided LiDAR-Based Panoptic Segmentation for Autonomous Driving

    Xian, Guozeng / Ji, Changyun / Zhou, Lin et al. | IEEE | 2023


    EFFICIENT TRANSFORMER-BASED PANOPTIC SEGMENTATION

    AICH ABHISHEK / SUH YUMIN / SCHULTER SAMUEL et al. | European Patent Office | 2025

    Free access

    EFFICIENT TRANSFORMER-BASED PANOPTIC SEGMENTATION

    AICH ABHISHEK / SUH YUMIN / SCHULTER SAMUEL et al. | European Patent Office | 2025

    Free access

    INFRASTRUCTURE ANALYSIS USING PANOPTIC SEGMENTATION

    SCHULTER SAMUEL / GARG SPARSH | European Patent Office | 2023

    Free access