The use of unmanned aerial vehicles (UAVs) for visible–thermal object detection has emerged as a powerful technique to improve accuracy and resilience in challenging contexts, including dim lighting and severe weather conditions. However, most existing research relies on Convolutional Neural Network (CNN) frameworks, limiting the application of the Transformer’s attention mechanism to mere fusion modules and neglecting its potential for comprehensive global feature modeling. In response to this limitation, this study introduces an innovative dual-modal object detection framework called Visual Prompt multi-modal Detection (VIP-Det) that harnesses the Transformer architecture as the primary feature extractor and integrates vision prompts for refined feature fusion. Our approach begins with the training of a single-modal baseline model to solidify robust model representations, which is then refined through fine-tuning that incorporates additional modal data and prompts. Tests on the DroneVehicle dataset show that our algorithm achieves remarkable accuracy, outperforming comparable Transformer-based methods. These findings indicate that our proposed methodology marks a significant advancement in the realm of UAV-based object detection, holding significant promise for enhancing autonomous surveillance and monitoring capabilities in varied and challenging environments.


    Access

    Download


    Export, share and cite



    Title :

    Drone-Based Visible–Thermal Object Detection with Transformers and Prompt Tuning


    Contributors:
    Rui Chen (author) / Dongdong Li (author) / Zhinan Gao (author) / Yangliu Kuai (author) / Chengyuan Wang (author)


    Publication date :

    2024




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    Unknown




    Misaligned Visible-Thermal Object Detection: A Drone-Based Benchmark and Baseline

    Song, Kechen / Xue, Xiaotong / Wen, Hongwei et al. | IEEE | 2024


    Drone patrol using thermal imaging for object detection

    Juang, Jih-Gau / Tu, Guan-Ting / Liao, Yu-Hsien et al. | SPIE | 2020


    DRONE IMAGE-BASED OBJECT DETECTION SYSTEM

    European Patent Office | 2025

    Free access

    DRONE IMAGE-BASED OBJECT DETECTION SYSTEM

    KIM YOON / LEE POOREUM / KIM HEECHEOL et al. | European Patent Office | 2023

    Free access

    DRONE VISIBLE AREA DISPLAY SYSTEM

    KOGA TOSHIO / SAKASHITA KAZUYA / TANIGAWA MASAYOSHI et al. | European Patent Office | 2024

    Free access