Policy gradient (PG) methods are popular reinforcement learning (RL) methods where a baseline is often applied to reduce the variance of gradient estimates. In multi-agent RL (MARL), although the PG theorem can be naturally extended, the effectiveness of multi-agent PG (MAPG) methods degrades as the variance of gradient estimates increases rapidly with the number of agents. In this paper, we offer a rigorous analysis of MAPG methods by, firstly, quantifying the contributions of the number of agents and agents’ explorations to the variance of MAPG estimators. Based on this analysis, we derive the optimal baseline (OB) that achieves the minimal variance. In comparison to the OB, we measure the excess variance of existing MARL algorithms such as vanilla MAPG and COMA. Considering using deep neural networks, we also propose a surrogate version of OB, which can be seamlessly plugged into any existing PG methods in MARL. On benchmarks of Multi-Agent MuJoCo and StarCraft challenges, our OB technique effectively stabilises training and improves the performance of multi-agent PPO and COMA algorithms by a significant margin. Code is released at https://github.com/morning9393/Optimal-Baseline-for-Multi-agent-Policy-Gradients.


    Access

    Download


    Export, share and cite



    Title :

    Settling the Variance of Multi-Agent Policy Gradients


    Contributors:
    Kuba, JG (author) / Wen, M (author) / Meng, L (author) / Gu, S (author) / Zhang, H (author) / Mguni, DH (author) / Wang, J (author) / Yang, Y (author)

    Publication date :

    2021-01-01


    Remarks:

    In: Advances in Neural Information Processing Systems 34 (NeurIPS 2021). (pp. pp. 13458-13470). NeurIPS Proceedings (2021)


    Type of media :

    Paper


    Type of material :

    Electronic Resource


    Language :

    English


    Classification :

    DDC:    629 / 006



    FACMAC: Factored multi−agent centralised policy gradients

    Peng, B / Rashid, T / Schroeder de Witt, CA et al. | BASE | 2021

    Free access

    BUILDING SETTLING DETECTION

    FAUCONNET AURÉLIEN / REBILLARD PASCAL / SIRIGU GÉRARD | European Patent Office | 2019

    Free access

    BUILDING SETTLING DETECTION

    FAUCONNET AURELIEN / REBILLARD PASCAL / SIRIGU GERARD | European Patent Office | 2017

    Free access


    Size distribution and settling velocity of cohesive sediments during settling

    Lau, Y. L. / Krishnappan, B. G. | British Library Online Contents | 1992