Timezone: »
Exploration has been a long standing problem in both model-based and model-free learning methods for sensorimotor control. There have been major advances in recent years demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of the current formulations get stuck when there are stochastic dynamics. In this paper, we propose a formulation for exploration inspired from the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to maximize the disagreement or variance of those ensembles. We show that this formulation works as well as other formulations in non-stochastic scenarios, and is able to explore better in scenarios with stochastic-dynamics. Further, we show that this objective can be leveraged to perform differentiable policy optimization. This leads to a sample efficient exploration policy. We show experiments on a large number of standard environments to demonstrate the efficacy of this approach. Furthermore, we implement our exploration algorithm on a real robot which learns to interact with objects completely from scratch. Project videos are in supplementary.
Author Information
Deepak Pathak (UC Berkeley)
Dhiraj Gandhi (Carnegie Mellon University Robotics Institute)
Abhinav Gupta (Carnegie Mellon University)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Self-Supervised Exploration via Disagreement »
Thu. Jun 13th 01:30 -- 04:00 AM Room Pacific Ballroom #39
More from the Same Authors
-
2020 : Neural Dynamic Policies for End-to-End Sensorimotor Learning »
Abhinav Gupta -
2021 Poster: PixelTransformer: Sample Conditioned Signal Generation »
Shubham Tulsiani · Abhinav Gupta -
2021 Spotlight: PixelTransformer: Sample Conditioned Signal Generation »
Shubham Tulsiani · Abhinav Gupta -
2020 Poster: Learning Robot Skills with Temporal Variational Inference »
Tanmay Shankar · Abhinav Gupta -
2018 Poster: Investigating Human Priors for Playing Video Games »
Rachit Dubey · Pulkit Agrawal · Deepak Pathak · Tom Griffiths · Alexei Efros -
2018 Oral: Investigating Human Priors for Playing Video Games »
Rachit Dubey · Pulkit Agrawal · Deepak Pathak · Tom Griffiths · Alexei Efros -
2017 Poster: Curiosity-driven Exploration by Self-supervised Prediction »
Deepak Pathak · Pulkit Agrawal · Alexei Efros · Trevor Darrell -
2017 Poster: Robust Adversarial Reinforcement Learning »
Lerrel Pinto · James Davidson · Rahul Sukthankar · Abhinav Gupta -
2017 Talk: Curiosity-driven Exploration by Self-supervised Prediction »
Deepak Pathak · Pulkit Agrawal · Alexei Efros · Trevor Darrell -
2017 Talk: Robust Adversarial Reinforcement Learning »
Lerrel Pinto · James Davidson · Rahul Sukthankar · Abhinav Gupta