Success-Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success

A widely used technique for improving policies is success conditioning, in which one collects trajectories, identifies those that achieve a desired outcome, and updates the policy to imitate the actions taken along successful trajectories. This principle appears under many names---rejection sampling with SFT, goal-conditioned RL, Decision Transformers---yet what optimization problem it solves, if any, has remained unclear. We prove that success conditioning exactly solves a trust-region optimization problem, maximizing policy improvement subject to a $\chi^2$ divergence constraint whose radius is determined automatically by the data. This yields an identity: relative policy improvement, the magnitude of policy change, and a quantity we call action-influence---measuring how random variation in action choices affects success rates---are exactly equal at every state. Success conditioning thus emerges as a conservative improvement operator. Exact success conditioning cannot degrade performance or induce dangerous distribution shift, but when it fails, it does so observably, by hardly changing the policy at all. We apply our theory to the common practice of return thresholding, showing this can amplify improvement, but at the cost of potential misalignment with the true objective.

Many of today's most important AI training methods share a deceptively simple recipe: have the model attempt a task many times, keep the attempts that succeeded, and train the model to imitate those successful attempts. This idea appears under many different names across the field, including both techniques popular in language modeling ("SFT + rejection sampling") and those popular in game playing and robotics ("Return Conditioned RL, Decision Transformers"). Practitioners favor these methods because they appear to skip the elaborate reinforcement learning procedures normally used to improve decision-making systems. So is "imitating success" actually solving a coherent problem, or just a trick that happens to work? We give this procedure a name — success conditioning — and formalize it as a mathematical operation. We then prove it is implicitly solving a well-defined optimization problem, one that improves behavior as much as it can while refusing to drift too far from what has already been tried. In other words, what looks like simple supervised learning on successful examples is actually a familiar kind of reinforcement learning update in disguise. This perspective yields a sharp characterization of when the method works, when it fails, and how to tell the difference before deployment.