A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -- from GANs and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player influence on others' updates. Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner's Dilemma. Although experimentally successful, we show that LOLA agents can exhibit 'arrogant' behaviour directly at odds with convergence. In fact, remarkably few algorithms have theoretical guarantees applying across all (n-player, non-convex) games. In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead. We prove that LookAhead converges locally to equilibria and avoids strict saddles in all differentiable games. SOS inherits these essential guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally.


    Access

    Download


    Export, share and cite



    Title :

    Stable Opponent Shaping in Differentiable Games


    Contributors:
    Letcher, A (author) / Foerster, J (author) / Balduzzi, D (author) / Rocktäschel, T (author) / Whiteson, S (author)

    Publication date :

    2019-05-01


    Remarks:

    In: Proceedings of the 7th International Conference on Learning Representations:ICLR 2019. ICLR: New Orleans, Louisiana,USA. (2019)


    Type of media :

    Paper


    Type of material :

    Electronic Resource


    Language :

    English



    Classification :

    DDC:    629



    The Mechanics of n-Player Differentiable Games

    Balduzzi, D / Racaniere, S / Martens, J et al. | BASE | 2018

    Free access

    Interactive computerized air combat opponent

    Hankins, W. W., III | NTRS | 1976


    Autonomous UAV Maneuvering Decisions by Refining Opponent Strategies

    Sun, Like / Qiu, Huaxin / Wang, Yangzhu et al. | IEEE | 2024


    Opponent piece mounting frame, glass assembly and vehicle

    WANG YIQI / WANG ZHIXIN / ZHENG KECHENG et al. | European Patent Office | 2024

    Free access