Timezone: »
Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch. Project videos and code are at https://pathak22.github.io/exploration-by-disagreement/
Author Information
Deepak Pathak (UC Berkeley)
Dhiraj Gandhi (Carnegie Mellon University Robotics Institute)
Abhinav Gupta (Carnegie Mellon University)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Oral: Self-Supervised Exploration via Disagreement »
Wed. Jun 12th 07:10 -- 07:15 PM Room Hall B
More from the Same Authors
-
2020 : Neural Dynamic Policies for End-to-End Sensorimotor Learning »
Abhinav Gupta -
2021 Poster: PixelTransformer: Sample Conditioned Signal Generation »
Shubham Tulsiani · Abhinav Gupta -
2021 Spotlight: PixelTransformer: Sample Conditioned Signal Generation »
Shubham Tulsiani · Abhinav Gupta -
2020 Poster: Learning Robot Skills with Temporal Variational Inference »
Tanmay Shankar · Abhinav Gupta -
2018 Poster: Investigating Human Priors for Playing Video Games »
Rachit Dubey · Pulkit Agrawal · Deepak Pathak · Tom Griffiths · Alexei Efros -
2018 Oral: Investigating Human Priors for Playing Video Games »
Rachit Dubey · Pulkit Agrawal · Deepak Pathak · Tom Griffiths · Alexei Efros -
2017 Poster: Curiosity-driven Exploration by Self-supervised Prediction »
Deepak Pathak · Pulkit Agrawal · Alexei Efros · Trevor Darrell -
2017 Poster: Robust Adversarial Reinforcement Learning »
Lerrel Pinto · James Davidson · Rahul Sukthankar · Abhinav Gupta -
2017 Talk: Curiosity-driven Exploration by Self-supervised Prediction »
Deepak Pathak · Pulkit Agrawal · Alexei Efros · Trevor Darrell -
2017 Talk: Robust Adversarial Reinforcement Learning »
Lerrel Pinto · James Davidson · Rahul Sukthankar · Abhinav Gupta