This paper investigates a population-based training regime based on game-theoretic principles called Policy-Spaced Response Oracles (PSRO). PSRO is general in the sense that it (1) encompasses well-known algorithms such as fictitious play and double oracle as special cases, and (2) in principle applies to general-sum, many-player games. Despite this, prior studies of PSRO have been focused on two-player zero-sum games, a regime where in Nash equilibria are tractably computable. In moving from two-player zero-sum games to more general settings, computation of Nash equilibria quickly becomes infeasible. Here, we extend the theoretical underpinnings of PSRO by considering an alternative solution concept, α-Rank, which is unique (thus faces no equilibrium selection issues, unlike Nash) and applies readily to general-sum, many-player settings. We establish convergence guarantees in several games classes, and identify links between Nash equilibria and α-Rank. We demonstrate the competitive performance of α-Rank-based PSRO against an exact Nash solver-based PSRO in 2-player Kuhn and Leduc Poker. We then go beyond the reach of prior PSRO applications by considering 3- to 5-player poker games, yielding instances where α-Rank achieves faster convergence than approximate Nash solvers, thus establishing it as a favorable general games solver. We also carry out an initial empirical validation in MuJoCo soccer, illustrating the feasibility of the proposed approach in another complex domain.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    A Generalized Training Approach for Multiagent Learning


    Beteiligte:
    Muller, P (Autor:in) / Omidshafiei, S (Autor:in) / Rowland, M (Autor:in) / Tuyls, K (Autor:in) / Pérolat, J (Autor:in) / Liu, S (Autor:in) / Hennes, D (Autor:in) / Marris, L (Autor:in) / Lanctot, M (Autor:in) / Hughes, E (Autor:in)

    Erscheinungsdatum :

    2020-01-01


    Anmerkungen:

    In: Proceedings of the 8th International Conference on Learning Representations, ICLR 2020. (pp. pp. 1-35). ICLR (2020)


    Medientyp :

    Paper


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Klassifikation :

    DDC:    629



    Multiagent Learning-Based Approach to Transit Assignment Problem

    Wahba, Mohammed / Shalaby, Amer | Transportation Research Record | 2005


    A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

    Lanctot, M / Zambaldi, V / Gruslys, A et al. | BASE | 2017

    Freier Zugriff

    Multiagent Learning-Based Approach to Transit Assignment Problem: A Prototype

    Wahba, Mohammed / Shalaby, Amer | Transportation Research Record | 2005