For all its successes, Reinforcement Learning (RL) still struggles to deliver formal guarantees on the closed-loop behavior of the learned policy. Among other things, guaranteeing the safety of RL with respect to safety-critical systems is a very active research topic. Some recent contributions propose to rely on projections of the inputs delivered by the learned policy into a safe set, ensuring that the system safety is never jeopardized. Unfortunately, it is unclear whether this operation can be performed without disrupting the learning process. This paper addresses this issue. The problem is analysed in the context of Q-learning and policy gradient techniques. We show that the projection approach is generally disruptive in the context of Q-learning though a simple alternative solves the issue, while simple corrections can be used in the context of policy gradient methods in order to ensure that the policy gradients are unbiased. The proposed results extend to safe projections based on robust MPC techniques.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Safe reinforcement learning via projection on a safe set: How to achieve optimality?



    Erscheinungsdatum :

    2020-01-01



    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Klassifikation :

    DDC:    629



    Routing using Safe Reinforcement Learning

    Nayak Seetanadi, Gautham / Årzén, Karl-Erik | BASE | 2020

    Freier Zugriff

    Routing Using Safe Reinforcement Learning

    Nayak Seetanadi, Gautham | BASE | 2020

    Freier Zugriff

    Safe Reinforcement Learning for Urban Driving using Invariably Safe Braking Sets

    Krasowski, Hanna / Zhang, Yinqiang / Althoff, Matthias | IEEE | 2022


    SAFE AND EFFICIENT REINFORCEMENT LEARNING ; Säker och effektiv reinforcement learning

    Magnusson, Björn / Forslund, Måns | BASE | 2019

    Freier Zugriff

    Safe Exploration in Model-Based Reinforcement Learning

    Cohen, Max / Belta, Calin | Springer Verlag | 2023