Multispectral pedestrian detection is an important task for many around-the-clock applications, since the visible and thermal modalities can provide complementary information especially under low light conditions. Due to the presence of two modalities, misalignment and modality imbalance are the most significant issues in multispectral pedestrian detection. In this paper, we propose MultiSpectral pedestrian DEtection TRansformer (MS-DETR) to fix above issues. MS-DETR consists of two modality-specific backbones and Transformer encoders, followed by a multi-modal Transformer decoder, and the visible and thermal features are fused in the multi-modal Transformer decoder. To well resist the misalignment between multi-modal images, we design a loosely coupled fusion strategy by sparsely sampling some keypoints from multi-modal features independently and fusing them with adaptively learned attention weights. Moreover, based on the insight that not only different modalities, but also different pedestrian instances tend to have different confidence scores to final detection, we further propose an instance-aware modality-balanced optimization strategy, which preserves visible and thermal decoder branches and aligns their predicted slots through an instance-wise dynamic loss. Our end-to-end MS-DETR shows superior performance on the challenging KAIST, CVC-14 and LLVIP benchmark datasets. The source code is available at https://github.com/YinghuiXing/MS-DETR.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    MS-DETR: Multispectral Pedestrian Detection Transformer With Loosely Coupled Fusion and Modality-Balanced Optimization


    Beteiligte:
    Xing, Yinghui (Autor:in) / Yang, Shuo (Autor:in) / Wang, Song (Autor:in) / Zhang, Shizhou (Autor:in) / Liang, Guoqiang (Autor:in) / Zhang, Xiuwei (Autor:in) / Zhang, Yanning (Autor:in)

    Erschienen in:

    Erscheinungsdatum :

    01.12.2024


    Format / Umfang :

    8488553 byte




    Medientyp :

    Aufsatz (Zeitschrift)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    PARAMETER ESTIMATION OF LOOSELY COUPLED TRANSFORMER

    DADRAS SARA / MALEK HADI / ELSHAER MOHAMED et al. | Europäisches Patentamt | 2017

    Freier Zugriff

    Loosely coupled GNSS and UWB with INS integration for indoor/outdoor pedestrian navigation

    Di Pietra V. / Dabove P. / Piras M. | BASE | 2020

    Freier Zugriff

    Toward Generalizable Multispectral Pedestrian Detection

    Chu, Fuchen / Cao, Jiale / Song, Zhanjie et al. | IEEE | 2024


    WRRT-DETR: Weather-Robust RT-DETR for Drone-View Object Detection in Adverse Weather

    Bei Liu / Jiangliang Jin / Yihong Zhang et al. | DOAJ | 2025

    Freier Zugriff

    LOOSELY-COUPLED LOCK-STEP CHAINING

    JOHNSON KERRY WAYNE / HOBBS CHRISTOPHER WILLIAM LEWIS / SHOOK PETER | Europäisches Patentamt | 2025

    Freier Zugriff