Zero-Shot Off-Policy Learning

Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions. This objective presents significant challenges, primarily due to the inherent distributional shift and value function overestimation bias. These issues become even more noticeable in zero-shot reinforcement learning, where an agent trained on reward-free data must adapt to new tasks at test time without additional training. In this work, we address the off-policy problem in a zero-shot setting by discovering a theoretical connection of successor measures to stationary density ratios. Using this insight, our algorithm can infer optimal importance sampling ratios, effectively performing a stationary distribution correction with an optimal policy for any task on the fly. We benchmark our method in motion tracking tasks on SMPL Humanoid, continuous control on ExoRL, and for the long-horizon OGBench tasks. Our technique seamlessly integrates into forward-backward representation frameworks and enables fast-adaptation to new tasks in a training-free regime. More broadly, this work bridges off-policy learning and zero-shot adaptation, offering benefits to both research areas.

Reinforcement learning agents often need large amounts of trial-and-error interaction to learn a new task, which can be expensive or unsafe in real-world settings such as robotics. A promising alternative is to train an agent once on a large collection of past experience and later ask it to solve new tasks without further practice. However, this can fail when the agent’s chosen behavior visits situations that were rare or missing in the original data, causing it to overestimate how good those behaviors are. We introduce Zero-Shot Off-Policy Learning, a method that helps an already trained agent adapt to new tasks using only offline data. The key idea is to estimate how much the agent’s intended behavior differs from the behavior represented in the dataset, and to use this estimate to correct the task objective during adaptation. Our method does not require new environment interactions or retraining the original model. Across robotic locomotion, maze navigation, manipulation, and humanoid motion-tracking tasks, this correction makes zero-shot adaptation more reliable. This work brings reinforcement learning closer to reusable, general-purpose agents that can safely adapt to new goals from existing experience.