Unsupervised Hierarchical Skill Discovery

We consider the problem of unsupervised skill segmentation and hierarchical structure discovery in reinforcement learning. While recent approaches have sought to segment trajectories into reusable skills or options, most rely on action labels, rewards, or handcrafted annotations, limiting their applicability. We propose a method that segments unlabelled trajectories into skills and induces a hierarchical structure over them using a grammar-based approach. The resulting hierarchy captures both low-level behaviours and their composition into higher-level skills. We evaluate our approach in high-dimensional, pixel-based environments, including Craftax and the full, unmodified version of Minecraft. Using metrics for skill segmentation, reuse, and hierarchy quality, we find that our method consistently produces more structured and semantically meaningful hierarchies than existing baselines. Furthermore, as a proof of concept, we demonstrate that these discovered hierarchies accelerate and stabilise learning on downstream reinforcement learning tasks.

It would be nice if computers could learn by watching others perform tasks. Unfortunately, most existing methods that try to do this rely on detailed annotations, such as labels describing what action is being taken, or explicit markers indicating where one skill ends and another begins (for example, signalling that a "walking" skill has finished and a "picking up an object" skill has started). This kind of supervision is very hard or impossible to get for most real-world tasks. Our method shows how we can do this by simply observing someone else's behaviour, like watching a video. It automatically breaks the video into smaller skills, then groups these skills into a larger hierarchy, much like words forming phrases in a language. This gives the computer both a local view of what is happening step by step and a higher-level view of how smaller behaviours combine into more complex ones. We tested our method in complicated games like Minecraft. We find that the resulting skill hierarchies are more structured and meaningful than those produced by existing methods. We also show that these learned hierarchies help a computer learn new tasks more efficiently.