Data from different modalities, such as infrared and visible images, can offer complementary information, and integrating such information can significantly enhance the capabilities of a system to perceive and recognize its surroundings. Thus, multi-modal object detection has widespread applications, particularly in challenging weather conditions like low-light scenarios. The core of multi-modal fusion lies in developing a reasonable fusion strategy, which can fully exploit the complementary features of different modalities while preventing a significant increase in model complexity. To this end, this paper proposes a novel lightweight cross-fusion module named Channel-Patch Cross Fusion (CPCF), which leverages Channel-wise Cross-Attention (CCA), Patch-wise Cross-Attention (PCA) and Adaptive Gating (AG) to encourage mutual rectification among different modalities. This process simultaneously explores commonalities across modalities while maintaining the uniqueness of each modality. Furthermore, we design a versatile intermediate fusion framework that can leverage CPCF to enhance the performance of multi-modal object detection. The proposed method is extensively evaluated on multiple public multi-modal datasets, namely FLIR, LLVIP, and DroneVehicle. The experiments indicate that our method yields consistent performance gains across various benchmarks and can be extended to different types of detectors, further demonstrating its robustness and generalizability. Our codes are available at https://github.com/Superjie13/CPCF_Multispectral.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Rethinking Self-Attention for Multispectral Object Detection


    Beteiligte:
    Hu, Sijie (Autor:in) / Bonardi, Fabien (Autor:in) / Bouchafa, Samia (Autor:in) / Prendinger, Helmut (Autor:in) / Sidibe, Desire (Autor:in)

    Erschienen in:

    Erscheinungsdatum :

    01.11.2024


    Format / Umfang :

    4881010 byte




    Medientyp :

    Aufsatz (Zeitschrift)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    OBJECT DETECTION WITH MULTISPECTRAL LIGHT

    BAKER SIMON / HUMM ERIK | Europäisches Patentamt | 2023

    Freier Zugriff


    Multispectral object detection with thermal imaging

    LOW YEW KWANG / TAYLOR RONALD M / EVERLY WARD K | Europäisches Patentamt | 2022

    Freier Zugriff

    Ret3D: Rethinking Object Relations for Efficient 3D Object Detection in Driving Scenes

    Wu, Y. / Zhang, D. / Zhang, L. et al. | BASE | 2022

    Freier Zugriff

    Concrete object recognition by multispectral systems

    Zlotnikov, K.A. | Tema Archiv | 1998