Transformers have shown promising progress in various visual object detection tasks, including monocular 2D/3D detection and surround-view 3D detection. More importantly, the attention mechanism in the Transformer model and the 3D information extraction in binocular stereo are both similarity-based. However, directly applying existing Transformer-based detectors to binocular stereo 3D object detection leads to slow convergence and significant precision drops. We argue that a key cause of that defect is that existing Transformers ignore the binocular-stereo-specific image correspondence information. In this paper, we explore the model design of Transformers in binocular 3D object detection, focusing particularly on extracting and encoding task-specific image correspondence information. To achieve this goal, we present TS3D, a Transformer-based Stereo-aware 3D object detector. In the TS3D, a Disparity-Aware Positional Encoding (DAPE) module is proposed to embed the image correspondence information into stereo features. The correspondence is encoded as normalized sub-pixel-level disparity and is used in conjunction with sinusoidal 2D positional encoding to provide the 3D location information of the scene. To enrich multi-scale stereo features, we propose a Stereo Preserving Feature Pyramid Network (SPFPN). The SPFPN is designed to preserve the correspondence information while fusing intra-scale and aggregating cross-scale stereo features. Our proposed TS3D achieves a 41.29% Moderate Car detection average precision on the KITTI test set and takes 88 ms to detect objects from each binocular image pair. It is competitive with advanced counterparts in terms of both precision and inference speed.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Transformer-Based Stereo-Aware 3D Object Detection From Binocular Images


    Contributors:
    Sun, Hanqing (author) / Pang, Yanwei (author) / Cao, Jiale (author) / Xie, Jin (author) / Li, Xuelong (author)

    Published in:

    Publication date :

    2024-12-01


    Size :

    2381398 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Binocular Stereo from Grey-Scale Images

    Nielsen, M. / Maas, R. / Niessen, W. J. et al. | British Library Online Contents | 1999


    Active binocular stereo

    Maru, N. / Nishikawa, A. / Miyazaki, F. et al. | IEEE | 1993


    Binocular Spherical Stereo

    Li, S. | Online Contents | 2008


    Binocular Spherical Stereo

    Shigang Li, | IEEE | 2008


    Occlusions and binocular stereo

    Geiger, D. / Ladendorf, B. / Yuille, A. L. | TIBKAT | 1993