Receding-Horizon Execution for Action Chunking in Offline-to-Online Reinforcement Learning

Action chunking improves offline-to-online reinforcement learning (RL) by using temporally extended actions for exploration and multi-step temporal-difference (TD) backups. However, a chunk length that accelerates value propagation can be too long for open-loop execution, since executing the full predicted chunk before replanning reduces reactivity and degrades performance. We study receding-horizon execution for Q-chunking policies with a fixed prediction horizon. We train a horizon-conditioned execution critic that evaluates prefixes of a predicted chunk under fixed and value-based execution rules. On a long-horizon manipulation task, receding-horizon execution substantially improves policies that fail under full open-loop execution but does not match a fixed short-horizon-trained actor or the best fixed execution horizon. Value-based horizon selection remains biased toward short horizons, and state-dependent selection remains an open problem.