Policy Transfer for Hierarchical Goal-Conditioned Reinforcement Learning

Goal-Conditioned Reinforcement Learning (GCRL) tackles the challenging problem of long-horizon, sparse-reward goal-reaching, often for robotic navigation or manipulation tasks in 2D or 3D spatial domains. Recent hierarchical approaches, relying on a two-level policy along with a graph of landmarks in goal space, have demonstrated reasonable asymptotic performance. However, existing algorithms suffer from poor sample efficiency due to too many uninformative landmarks and the inability to transfer low-level behaviour between related tasks. We instead claim that transferring a pre-trained low-level policy between environments can significantly improve landmark generation, sample efficiency and even success rates. We introduce an algorithm PROMO, which uses a pre-trained low-level policy and a learned reachability model to generate landmarks that give access to large novel (previously unreachable) regions of goal space whilst remaining reachable from the previous landmark. Whereas previous works incorporate the idea of reachability/novelty only implicitly or using heuristics, we explicitly model reachability using a structured neural architecture. We demonstrate 3-4x improvements in sample efficiency over existing state-of-the-art methods on the challenging robotics tasks of AntMaze and Reacher3D, with the mild overhead of one-time policy pre-training. In addition, our method achieves superior success rates across all environments, as well as better training stability and much fewer, more informative landmarks.