Scalable Option Learning in High-Throughput Environments

Hierarchical reinforcement learning (RL) has the potential to enable effective decision-making over long timescales. Existing approaches, while promising, have yet to realize the benefits of large-scale training. In this work, we identify and solve several key challenges in scaling online hierarchical RL to high-throughput environments. We propose Scalable Option Learning (SOL), a highly scalable hierarchical policy gradient algorithm which achieves a ~35x higher throughput compared to existing hierarchical methods. To demonstrate SOL's performance and scalability, we train hierarchical agents using 30 billion frames of experience on the complex game of NetHack, significantly surpassing flat agents and demonstrating positive scaling trends. We also validate SOL on MiniHack and Mujoco environments, showcasing its general applicability.

Training AI agents to solve tasks which require many decision steps is difficult. Hierarchical systems with both a high-level and a low-level agent can make this easier by breaking the task into many subtasks that are easier to solve. However, such systems have not yet been scaled using large amounts of data and compute. We identify reasons why scaling these systems is difficult and propose a new way to train hierarchical agents using lots of data and compute, which is around 35 times faster than prior methods. We apply our method to a very complicated video game which requires many decision steps, and show that performance keeps getting better as we apply more data and compute. We also show that our method works in other settings, not only this video game. We share our code so that the community can build upon our work.