3D semantic segmentation is a key technology for intelligent vehicles. Recently, great efforts have been made to achieve accurate and robust 3D semantic segmentation results through LiDAR-camera fusion, due to the complementary information of images like color and texture. However, current LiDAR-camera fusion methods still lag behind LiDAR-only methods in complex and varied scenarios containing sparse and similar geometric features. Three main factors limit their performance: sparse mappings between two modalities, the contribution of different modality features, and the gap between results of two modalities. Addressing that, we present a multi-phase LiDAR-camera fusion method, which comprehensively fuses information of two modalities in three phases: first, a context sampler block is proposed to fuse low-level image contextual features into LiDAR points; second, a cross dimensional fusion block is designed to implement the interaction of shared features for hierarchical supervision while retaining private features for jointly refining their predictions; third, a modality perceptual conformance loss is induced to make them learn from each other in an explicit manner by minimizing the gap between two modality predictions. Quantitative and qualitative evaluations with state-of-the-art (SOTA) methods are performed on urban dataset SemanticKITTI, off-road dataset RELLIS-3D and our unstructured test set. The experimental results validate the competitiveness of our method in improving accuracy and scene-adaptivity.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Robust 3D Semantic Segmentation Based on Multi-Phase Multi-Modal Fusion for Intelligent Vehicles


    Contributors:
    Ni, Peizhou (author) / Li, Xu (author) / Xu, Wang (author) / Kong, Dong (author) / Hu, Yue (author) / Wei, Kun (author)

    Published in:

    Publication date :

    2024-01-01


    Size :

    5916604 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English






    MULTI-MODAL SEGMENTATION NETWORK FOR ENHANCED SEMANTIC LABELING IN MAPPING

    WIDJAJA SERGI ADIPRAJA / SHARMA DHANANJAI / LIONG VENICE ERIN B | European Patent Office | 2023

    Free access

    Multi-modal segmentation network for enhanced semantic labeling in mapping

    SERGI ADIPRAJA WIDJAJA / DHANANJAI SHARMA / VENICE ERIN BAYLON LIONG | European Patent Office | 2025

    Free access