Abstract Data augmentation is an important pre‐processing step for object detection in 2D image and 3D point clouds. However, studies on multimodal data augmentation are extremely limited compared to single‐modal work. Moreover, simultaneously ensuring consistency and rationality when pasting both image and point cloud samples is a major challenge in multimodal methods. In this study, a novel multimodal data augmentation method based on ground truth sampling (GT sampling) is proposed for generating content‐rich synthetic scenes. A GT database and scene ground database based on the raw training set is initially built, following which the context of the image and point cloud is used to guide the paste location and filtering strategy of the GT samples. The proposed method can avoid the cluttered features caused by random pasting of samples; the image context information can help the model to learn the correlation between the object and the environment more comprehensively, and the point cloud context information can reduce occlusion in the case of long‐distance objects. The effectiveness of the proposed strategy is demonstrated on the publicly available KITTI dataset. Utilizing the multimodal 3D detector MVXNet as an implementation tool, our experiments evaluate different superimposition strategies ranging from context‐free sample pasting methods to context‐guided new training scenes. In comparison with existing GT sampling methods, our method exhibits a relative performance improvement of 15% on benchmark datasets. In ablation studies, our sample pasting strategy achieves a +2.81% gain compared with previous work. In conclusion, considering the multimodal context of modelled objects is crucial for placing them in the correct environment.


    Access

    Download


    Export, share and cite



    Title :

    Context‐guided ground truth sampling for multi‐modality data augmentation in autonomous driving


    Contributors:
    Peicheng Shi (author) / Heng Qi (author) / Zhiqiang Liu (author) / Aixi Yang (author)


    Publication date :

    2023




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    Unknown




    Context‐guided ground truth sampling for multi‐modality data augmentation in autonomous driving

    Shi, Peicheng / Qi, Heng / Liu, Zhiqiang et al. | Wiley | 2023

    Free access

    MULTI-MODALITY DATA AUGMENTATION ENGINE TO IMPROVE RARE DRIVING SCENARIO DETECTION FOR VEHICLE SENSORS

    TANG LUAN / JIANG SHEPARD / YUAN PENG et al. | European Patent Office | 2024

    Free access

    Cooperative Raw Sensor Data Fusion for Ground Truth Generation in Autonomous Driving

    Ye, Egon / Spiegel, Philip / Althoff, Matthias | IEEE | 2020



    GROUND TRUTH DATA GENERATION FOR DEEP NEURAL NETWORK PERCEPTION IN AUTONOMOUS DRIVING APPLICATIONS

    TILMAN WEKEL / JOACHIM PEHSERL / JACOB MEYER et al. | European Patent Office | 2022

    Free access