Chunk-Guided Q-Learning

In offline reinforcement learning (RL), single-step temporal-difference (TD) learning can suffer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mitigate this by backing up over multiple steps, but can introduce suboptimality by restricting the policy class to open-loop action sequences. To resolve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.

In reinforcement learning, an agent learns by using an critic to estimate the value of each action. Because rewards often arrive only at the end of a long task, the critic has to estimate these values based on its own predictions of what comes next. This bootstrapping is a long-standing challenge: small estimation errors compound across long sequences, and the value estimates can drift far from the truth. A common fix is to train the critic over chunks of multiple actions at once, reducing the number of steps over which errors accumulate. This is like reading a map at a coarser scale: the overall route becomes clearer, but individual turns are lost. The critic becomes more stable but can no longer judge actions one by one. We instead combine the two: a fine-grained critic that evaluates actions one at a time, anchored by a chunk-based critic toward more stable estimates. One might expect this combination to land somewhere between the two; surprisingly, both in theory and in experiments, it outperforms either alone. Our findings point to a new direction for stabilizing value learning on long-horizon tasks such as robotic manipulation and multi-step planning.