In autonomous driving, monocular sequences contain lots of information. Monocular depth estimation, camera ego-motion estimation and optical flow estimation in consecutive frames are high-profile concerns recently. By analyzing tasks above, pixels in the middle frame are modeled into three parts: the rigid region, the non-rigid region, and the occluded region. In joint unsupervised training of depth and pose, we can segment the occluded region explicitly. The occlusion information is used in unsupervised learning of depth, pose and optical flow, as the image reconstructed by depth-pose and optical flow will be invalid in occluded regions. A less-than-mean mask is designed to further exclude the mismatched pixels interfered with by motion or illumination change in the training of depth and pose networks. This method is also used to exclude some trivial mismatched pixels in the training of the optical flow network. Maximum normalization is proposed for depth smoothness term to restrain depth degradation in textureless regions. In the occluded region, as depth and camera motion can provide more reliable motion estimation, they can be used to instruct unsupervised learning of optical flow. Our experiments in KITTI dataset demonstrate that the model based on three regions, full and explicit segmentation of the occlusion region, the rigid region, and the non-rigid region with corresponding unsupervised losses can improve performance on three tasks significantly. The source code is available at: https://github.com/guangmingw/DOPlearning.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Unsupervised Learning of Depth, Optical Flow and Pose With Occlusion From 3D Geometry


    Contributors:
    Wang, Guangming (author) / Zhang, Chi (author) / Wang, Hesheng (author) / Wang, Jingchuan (author) / Wang, Yong (author) / Wang, Xinlei (author)


    Publication date :

    2022-01-01


    Size :

    7737772 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Unsupervised Learning of Optical Flow With Non-Occlusion From Geometry

    Wang, Guangming / Ren, Shuaiqi / Wang, Hesheng | IEEE | 2022


    Unsupervised Learning of Depth Estimation and Camera Pose With Multi-Scale GANs

    Xu, Yufan / Wang, Yan / Huang, Rui et al. | IEEE | 2022


    Unsupervised Deep Learning of Depth, Ego-Motion, and Optical Flow from Stereo Images

    Yang, Delong / Luo, Zhaohui / Shang, Peng et al. | IEEE | 2021


    Multiple people tracking and pose estimation with occlusion estimation

    Huo, F. / Hendriks, E. A. | British Library Online Contents | 2012


    Sparse Occlusion Detection with Optical Flow

    Ayvaci, A. / Raptis, M. / Soatto, S. | British Library Online Contents | 2012