COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space

Unsupervised skill discovery (USD) aims to learn diverse behaviors without reward functions, but often results in task-irrelevant or hazardous behaviors due to uniform exploration. Guided skill discovery (GSD) addresses this issue by incorporating human intent to focus exploration on meaningful regions. However, existing GSD methods typically require training additional guidance models, and rely on pre-defined rules or expert demonstration, which can be ineffective under sparse, online-collected human feedback. To overcome this, we propose COLLIE, a GSD framework that leverages dense unsupervised data to construct a semantically coherent skill latent space. This latent space is well-structured, enabling reliable guidance with sparse online feedback. Moreover, its semantic coherence property enables training-free construction of guidance signals, eliminating the need for additional model training beyond skill learning. Theoretical analysis justifies the effectiveness of our training-free guidance signal, while experiments across diverse state-based and pixel-based tasks show that COLLIE learns diverse, human-aligned skills, avoids hazardous behaviors, and achieves superior downstream performance with minimal human feedback.

Teaching machines to discover useful behaviors on their own is challenging because they often waste time exploring irrelevant or even dangerous actions. Human guidance can help, but existing approaches require building extra prediction models or relying on rigid rules and expert demonstrations, which are impractical when human feedback is scarce and collected in real time. We propose COLLIE, a framework that first lets the machine learn from large amounts of unlabeled experience to build a well-organized "behavior map". Because this map is structured and meaningful, even a small amount of online human feedback can reliably guide exploration toward safe and useful behaviors. Notably, COLLIE constructs guidance signals directly from this map without training any additional models, making it both efficient and easy to deploy. Our theoretical analysis and experiments across pixel and state-based tasks show that COLLIE learns diverse, human-aligned skills while avoiding hazards. It also achieves strong performance on downstream tasks with minimal human supervision, making autonomous skill learning safer and more practical for real-world applications.