In the field of real-time autonomous decision-making for Unmanned Combat Aerial Vehicles (UCAVs), reinforcement learning is widely used to enhance their decision-making capabilities in high-dimensional spaces. These enhanced capabilities allow UCAVs to better respond to the maneuvers of various opponents, with the win rate often serving as the primary optimization metric. However, relying solely on the terminal outcome of victory or defeat as the optimization target, but without incorporating additional rewards throughout the process, poses significant challenges for reinforcement learning due to the sparse reward structure inherent in these scenarios. While algorithms enhanced with densely distributed artificial rewards show potential, they risk deviating from the primary objectives. To address these challenges, we introduce a novel approach: the homotopy-based soft actor–critic (HSAC) method. This technique gradually transitions from auxiliary tasks enriched with artificial rewards to the main task characterized by sparse rewards through homotopic paths. We demonstrate the consistent convergence of the HSAC method and its effectiveness through deployment in two distinct scenarios within a 3D air combat game simulation: attacking horizontally flying UCAVs and a combat scenario involving two UCAVs. Our experimental results reveal that HSAC significantly outperforms traditional algorithms, which rely solely on using sparse rewards or those supplemented with artificially aided rewards.


    Access

    Download


    Export, share and cite



    Title :

    Enhancing Automated Maneuvering Decisions in UCAV Air Combat Games Using Homotopy-Based Reinforcement Learning


    Contributors:
    Yiwen Zhu (author) / Yuan Zheng (author) / Wenya Wei (author) / Zhou Fang (author)


    Publication date :

    2024




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    Unknown




    Automated maneuvering decisions for air-to-air combat

    AUSTIN, FRED / CARBONE, GIRO / FALCO, MICHAEL et al. | AIAA | 1987


    Automated maneuvering decisions for air-to-air combat

    Austin, Fred / Carbone, Giro / Falco, Michael et al. | NTRS | 1987


    Manual-Based Automated Maneuvering Decisions for Air-to-Air Combat

    Yang, Kwangjin / Kim, Songhyon / Lee, Younggun et al. | AIAA | 2024

    Free access

    CERBERUS UCAV - Unmanned Combat Aerial Vehicle

    Kats, M. / Konevsky, A. / Buium, T. et al. | British Library Conference Proceedings | 2010


    A sample selection mechanism for multi-UCAV air combat policy training using multi-agent reinforcement learning

    YAN, Zihui / LIANG, Xiaolong / HOU, Yueqi et al. | Elsevier | 2025

    Free access