Timezone: »

 
Oral
An Investigation of Model-Free Planning
Arthur Guez · Mehdi Mirza · Karol Gregor · Rishabh Kabra · Sebastien Racaniere · Theophane Weber · David Raposo · Adam Santoro · Laurent Orseau · Tom Eccles · Greg Wayne · David Silver · Timothy Lillicrap

Tue Jun 11 02:00 PM -- 02:20 PM (PDT) @ Hall B
in Deep RL 2 »

The field of reinforcement learning (RL) is facing increasingly challenging domains with combinatorial complexity. For an RL agent to address these challenges, it is essential that it can plan effectively. Prior work has typically utilized an explicit model of the environment, combined with a specific planning algorithm (such as tree search). More recently, a new family of methods have been proposed that learn how to plan, by providing the structure for planning via an inductive bias in the function approximator (such as a tree structured neural network), trained end-to-end by a model-free RL algorithm. In this paper, we go even further, and demonstrate empirically that an entirely model-free approach, without special structure beyond standard neural network components such as convolutional networks and LSTMs, can learn to exhibit many of the characteristics typically associated with a model-based planner. We measure our agent's effectiveness at planning in terms of its ability to generalize across a combinatorial and irreversible state space, its data efficiency, and its ability to utilize additional thinking time. We find that our agent has many of the characteristics that one might expect to find in a planning algorithm. Furthermore, it exceeds the state-of-the-art in challenging combinatorial domains such as Sokoban and outperforms other model-free approaches that utilize strong inductive biases toward planning.

[ Video] 

Author Information

Arthur Guez (Google DeepMind)
Mehdi Mirza (DeepMind)
Karol Gregor (DeepMind)
Rishabh Kabra (DeepMind)
Sebastien Racaniere (DeepMind)
Theophane Weber (DeepMind)
David Raposo (DeepMind)
Adam Santoro (DeepMind)
Laurent Orseau (DeepMind)
Tom Eccles (DeepMind)
Greg Wayne (DeepMind)
David Silver (Google DeepMind)
Timothy Lillicrap (Google DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors