Good Experience Maximization

Post-training language models typically follows one of two paradigms: reinforcement learning (RL) and supervised fine-tuning (SFT). Although typically thought of as different camps, we show that both paradigms can be viewed as target distribution matching. We use this perspective to analyze the tension between policy gradient and buffer-based SFT and establish that a good post-training target should preserve past successes while continuing to incorporate newly-discovered ones. We introduce Good Experience Maximization (GEM), a simple framework that resolves this tension by training the policy toward a mixture of historical and newly generated successes. This construction is amenable to a variety of simple algorithmic choices, is agnostic to the sampling process that populates the buffer and naturally subsumes several existing post-training methods as special cases. We prove that the GEM target yields globally optimal expected reward at convergence under realizability, and we derive a flow-based update that guarantees one-step policy improvement. This is a distinctively different update to SFT and relies on constructing a straight line "path" toward the target distribution. Empirically both the "flow" and the cross entropy version of GEM perform on par with leading post-training methods and out-perform it in sparse reward settings.