To accumulate knowledge and improve its policy of behaviour, a reinforcement learning agent can learn ‘off-policy’ about policies that differ from the policy used to generate its experience. This is important to learn counterfactuals, or because the experience was generated out of its own control. However, off-policy learning is non-trivial, and standard reinforcementlearning algorithms can be unstable and divergent. In this paper we discuss a novel family of off-policy prediction algorithms which are convergent by construction. The idea is to frst learn on-policy about the data-generating behaviour, and then bootstrap an off-policy value estimate on this onpolicy estimate, thereby constructing a value estimate that is partially off-policy. This process can be repeated to build a chain of value functions, each time bootstrapping a new estimate on the previous estimate in the chain. Each step in the chain is stable and hence the complete algorithm is guaranteed to be stable. Under mild conditions this comes arbitrarily close to the off-policy TD solution when we increase the length of the chain. Hence it can compute the solution even in cases where off-policy TD diverges. We prove that the proposed scheme is convergent and corresponds to an iterative decomposition of the inverse key matrix. Furthermore it can be interpreted as estimating a novel objective – that we call a ‘k-step expedition’ – of following the target policy for fnitely many steps before continuing indefnitely with the behaviour policy. Empirically we evaluate the idea on challenging MDPs such as Baird’s counter example and observe favourable results.


    Access

    Download


    Export, share and cite



    Title :

    Chaining Value Functions for Off-Policy Learning



    Publication date :

    2022-02-22


    Remarks:

    In: Sycara, Katia, (ed.) Proceedings of the AAAI 2022 Conference: The 36th AAAI Conference on Artificial Intelligence. (pp. pp. 8187-8195). Association for the Advancement of Artificial Intelligence (AAAI): Virtual conference. (2022)


    Type of media :

    Paper


    Type of material :

    Electronic Resource


    Language :

    English


    Classification :

    DDC:    629



    RIDE CHAINING

    LAMBERT CHRIS / GREEN LOGAN / YOO FRANK TAEHYUN et al. | European Patent Office | 2020

    Free access

    Ride chaining

    LAMBERT CHRIS / GREEN LOGAN / YOO FRANK TAEHYUN et al. | European Patent Office | 2019

    Free access

    RIDE CHAINING

    LAMBERT CHRIS / GREEN LOGAN / YOO FRANK TAEHYUN et al. | European Patent Office | 2021

    Free access

    Ride chaining

    LAMBERT CHRIS / GREEN LOGAN / YOO FRANK TAEHYUN et al. | European Patent Office | 2019

    Free access

    Ride chaining

    LAMBERT CHRIS / GREEN LOGAN / YOO FRANK TAEHYUN et al. | European Patent Office | 2023

    Free access