Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL

Unsupervised Reinforcement Learning (URL) aims to pre-train scalable, skill-conditioned policies without extrinsic rewards, serving as a foundation for downstream control tasks. Despite recent progress, we argue that current off-policy URL methods are limited by two critical, overlooked bottlenecks: (1) non-stationarity skill semantic and (2) brittle generalization. To address these challenges, we propose GenDa (Generalizable Data-efficient Agent), a unified framework for robust unsupervised reinforcement learning. First, we introduce a skill relabeling mechanism to mitigate non-stationarity and significantly improve sample efficiency for pretraining. Second, we propose a Complementary Information Bottleneck(CIB), encouraging the learned skill policy to focus on ego-centric features and become robust to distribution shifts for downstream tasks. Through various experiments, we demonstrate that GenDa significantly enhances the scalability of URL with superior generality and sample efficiency. Our source codes are available in the supplementary materials.

Robots and other autonomous agents can learn by trying things on their own, before we give them a specific task. In this stage, an agent might discover reusable skills such as walking, turning, or exploring. The challenge is that today’s methods can be slow and fragile. They often reuse old practice data even after the meaning of each skill has changed, and they may accidentally memorize details such as starting position or background instead of learning behavior that transfers. We introduce GenDa, a training method designed to make self-practice more efficient and reusable. GenDa relabels old experiences according to the agent’s current understanding, so past data remains useful rather than misleading. It also filters what the agent’s controller sees, encouraging it to rely on skill-relevant body information instead of irrelevant context. In simulated robot benchmarks, GenDa learns more diverse skills with fewer interactions and transfers them better to new tasks and settings. This can help build agents that adapt more easily after self-directed practice.