Hierarchical Goal Abstractions via Learned Subset Relations

In self-supervised goal-conditioned reinforcement learning (RL) without external rewards, goals are typically specified by observations sampled from experience. However, depending on the observation structure, such a fixed representation of goals may be either too concrete (requiring exact pixel-level matches) or too abstract (involving ambiguous observations). Here we propose the construction of hierarchical latent goal spaces that integrate both concrete and abstract goals. To this end, we use an energy function to learn a partially ordered space, in which a subset relation between observations naturally induces a hierarchy from concrete to abstract goals. This representation enables agents to disambiguate specific states while also generalizing to shared concepts. In experiments on navigation and robotic manipulation, agents trained with our hierarchical goal space achieve higher task success and greater generalization to novel tasks compared to agents limited to purely observational goals.

When teaching artificial intelligence (AI) to achieve goals on its own, researchers often use past experiences as targets. However, these target goals can be flawed: they are sometimes too rigid, requiring the AI to match exact visual details perfectly, or too vague, leaving the AI confused about what it actually needs to accomplish To solve this, we developed a method that helps AI systems understand goals on multiple levels. We built a system that automatically organizes targets into a hierarchy, ranging from highly specific details to broad, abstract concepts. This flexible structure allows the AI to grasp both the big picture and the fine details simultaneously. By learning to view goals in this structured way, the AI can apply its general knowledge to new situations while clearly distinguishing between specific tasks. In our tests involving navigation and robotic movement, AI agents using our hierarchical approach were far more successful and adapted better to new, unseen challenges compared to traditional methods.