Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually simple and principled application of RL methods. However real-world settings are many-agent, and agent interactions are complex mixtures of common-interest and competitive aspects. We consider Diplomacy, a 7-player board game designed to accentuate dilemmas resulting from many-agent interactions. It also features a large combinatorial action space and simultaneous moves, which are challenging for RL algorithms. We propose a simple yet effective approximate best response operator, designed to handle large combinatorial action spaces and simultaneous moves. We also introduce a family of policy iteration methods that approximate fictitious play. With these methods, we successfully apply RL to Diplomacy: we show that our agents convincingly outperform the previous state-of-the-art, and game theoretic equilibrium analysis shows that the new process yields consistent improvements.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Learning to Play No-Press Diplomacy with Best Response Policy Iteration


    Beteiligte:
    Anthony, TW (Autor:in) / Eccles, T (Autor:in) / Tacchetti, A (Autor:in) / Kramár, J (Autor:in) / Gemp, IM (Autor:in) / Hudson, TC (Autor:in) / Porcel, N (Autor:in) / Lanctot, M (Autor:in) / Pérolat, J (Autor:in) / Everett, R (Autor:in)

    Erscheinungsdatum :

    2020-12-12


    Anmerkungen:

    In: Advances in Neural Information Processing Systems 33 pre-proceedings (NeurIPS 2020). NeurIPS (2020) (In press).


    Medientyp :

    Paper


    Format :

    Elektronische Ressource


    Sprache :

    Englisch


    Klassifikation :

    DDC:    629



    Comparison of Value Iteration, Policy Iteration and Q-Learning for solving Decision-Making problems

    Hamadouche, Mohand / Dezan, Catherine / Espes, David et al. | IEEE | 2021


    Manipulating the Distributions of Experience used for Self-Play Learning in Expert Iteration

    Soemers, Dennis / Piette, Eric / Stephenson, Matthew et al. | BASE | 2020

    Freier Zugriff

    Trajectory-Based Modified Policy Iteration

    R. Sharma / M. Gopal | BASE | 2007

    Freier Zugriff


    AUTONOMOUS SOARING POLICY INITIALIZATION THROUGH VALUE ITERATION

    Rothaupt, Benjamin J. / Notter, Stefan / Fichter, Walter | TIBKAT | 2021