OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

Generative control policies (GCPs), such as diffusion- and flow-based control policies, have proved effective parameterizations for robot learning. This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient algorithm for finetuning GCPs that maintains off-policy critics to maximize data reuse and propagate policy gradients through the full generative process of the policy via a modified PPO objective, using critics as the terminal reward. OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control. To our knowledge, it is also the only method that can fine-tune poorly-initialized behavior cloning policies to near full task-success with no expert data in the online replay buffer, and does so with few task-specific hyperparameter tuning. Through extensive investigations, we demonstrate that OGPO drastically outperforms alternative methods on policy steering and learning residual corrections, and identify the key mechanisms behind its performance. We further introduce practical stabilization tricks, including success-buffer regularization and two-sided conservative advantages to mitigate critic over-exploitation across state- and pixel-based settings. Beyond proposing OGPO, we conduct a systematic empirical study of GCP finetuning, identifying the stabilizing mechanisms and failure modes that govern successful off-policy full-policy improvement.

OGPO: Teaching robots to improve themselves with less help Robots can learn manipulation skills, like inserting a peg or opening a door, by copying human demonstrations. But this kind of learning is fragile: a small change in the task or environment often makes the robot fail, and fixing it usually means collecting lots more human examples. We asked whether a robot could instead improve on its own, through trial and error, without all that extra human effort. To do this, we built a method called OGPO. Modern robots often plan their movements through a step-by-step "imagination" process before acting. The catch is that practicing in the real world is slow and expensive, while imagining possible actions is fast and cheap. OGPO exploits this gap: it reuses every real-world attempt efficiently to judge which actions are good, then does most of its trial-and-error inside the robot's cheap imagination. The result is a method that learns reliably across many difficult tasks with little hand-tuning, and can even rescue a robot that started out failing most of the time. This points toward robots that keep getting better after deployment, with far less human supervision.