Timezone: »

 
Poster
Generalised Policy Improvement with Geometric Policy Composition
Shantanu Thakoor · Mark Rowland · Diana Borsa · Will Dabney · Remi Munos · Andre Barreto

Thu Jul 21 03:00 PM -- 05:00 PM (PDT) @ Hall E #932
in Poster Session 3 »

We introduce a method for policy improvement that interpolates between the greedy approach of value-based reinforcement learning (RL) and the full planning approach typical of model-based RL. The new method builds on the concept of a geometric horizon model (GHM, also known as a \gamma-model), which models the discounted state-visitation distribution of a given policy. We show that we can evaluate any non-Markov policy that switches between a set of base Markov policies with fixed probability by a careful composition of the base policy GHMs, without any additional learning. We can then apply generalised policy improvement (GPI) to collections of such non-Markov policies to obtain a new Markov policy that will in general outperform its precursors. We provide a thorough theoretical analysis of this approach, develop applications to transfer and standard RL, and empirically demonstrate its effectiveness over standard GPI on a challenging deep RL continuous control task. We also provide an analysis of GHM training methods, proving a novel convergence result regarding previously proposed methods and showing how to train these models stably in deep RL settings.

Keywords: Reinforcement Learning

Author Information

Shantanu Thakoor (DeepMind)
Mark Rowland (DeepMind)
Diana Borsa (DeepMind)
Will Dabney (DeepMind)
Remi Munos (DeepMind)
Andre Barreto (DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors