The Reflective Explorer: Online Meta-Exploration from Offline Data in Visual Tasks with Sparse Rewards

Reinforcement learning is difficult to apply to real world problems due to high sample complexity, the need to adapt to regular distribution shifts, often encountered in the real world, and the complexities of learning from high-dimensional inputs, such as images. Over the last several years meta-learning has emerged as a promising approach to tackle these problems by explicitly training an agent to quickly adapt to novel tasks. However, such methods still require huge amounts of data during training are are difficult to optimize in high-dimensional domains. One potential solution is to consider offline or batch meta-learning - learning from existing datasets without additional environment interactions during training. In this work we develop the first offline meta-learning algorithm that operates from images in tasks with sparse rewards. Our approach has three main components: a novel strategy to construct meta-exploration trajectories from offline data, a deep variational filter training and latent offline model-free policy optimization. We show that our method completely solves a realistic meta-learning task involving robot manipulation, while naive combinations of meta-learning and offline algorithms significantly under-perform.