Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/

Many AI agents struggle with tasks that require a long sequence of decisions, such as navigating through a large maze or manipulating several objects in the right order. A common strategy is to break the task into smaller goals, but existing methods often choose only one intermediate goal and use separate systems for planning and acting. This can make the agent lose sight of the final goal or fail when the intermediate goal is not useful. We propose Chain-of-Goals Hierarchical Policy (CoGHP), a method that helps an agent plan through a sequence of intermediate goals before choosing its next action. Given the current state and the final goal, CoGHP generates a “chain of goals,” where each goal provides guidance for the next step. This is inspired by how step-by-step reasoning can help solve complex problems. By using one unified model to generate both intermediate goals and actions, CoGHP keeps the final goal in mind throughout the decision process. In navigation and robotic manipulation tasks, this approach helps agents solve longer and more complex tasks more reliably than strong existing methods.