Action-Sufficient Goal Representations

Hierarchical policies in offline goal-conditioned reinforcement learning (GCRL) addresses long-horizon tasks by decomposing control into high-level subgoal planning and low-level action execution. A critical design choice in such architectures is the goal representation—the compressed encoding of goals that serves as the interface between these levels. Existing approaches commonly derive goal representations while learning value functions, implicitly assuming that preserving information sufficient for value estimation is adequate for optimal control. We show that this assumption can fail, even when the value estimation is exact, as such representations may collapse goal states that need to be differentiated for action learning. To address this, we introduce an information-theoretic framework that defines action sufficiency, a condition on goal representations necessary for optimal action selection. We prove that value sufficiency does not imply action sufficiency and empirically verify that the latter is more strongly associated with control success in a discrete environment. We further demonstrate that standard log-loss training of low-level policies naturally induces action-sufficient representations. Our experimental results a popular benchmark demonstrate that our actor-derived representations consistently outperform representations learned via value estimation.

When an AI agent tackles a multi-step task — like a robot rearranging objects on a table — it often breaks the task into smaller subgoals. A "planner" decides what intermediate subgoals to aim for, and a "controller" executes the actions to reach each one. These two parts communicate through a compressed description of each subgoal. The standard approach builds these descriptions by predicting "how good is it to be here?" But this has a subtle problem: two states may look equally good while requiring completely opposite actions. We introduce a new criterion called action sufficiency: a good subgoal description must preserve enough information for the controller to know what action to take. We mathematically prove that the standard "how good is it?" approach can fail to satisfy this. Instead, we propose deriving the description directly from the controller's own behavior — with a theoretical guarantee that the result is action-sufficient. Across diverse robotic tasks, our method substantially outperforms the standard approach, especially when agents must learn from raw images. More broadly, our work clarifies what makes a subgoal description useful for AI control, offering a principled foundation for designing hierarchical AI systems.