Efficient RL Training for LLMs with Experience Replay

While Experience Replay—the practice of storing rollouts and reusing them multiple times during training—is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading -- and in some cases even improving -- final model performance, while preserving policy entropy.

Training large language models often involves asking the model to generate many candidate answers, evaluating those answers, and then using the results to improve the model. A common belief is that these generated examples should always be fresh: they should come from the latest version of the model being trained. This can be very expensive, because generating new examples requires a lot of compute. We study whether models can instead reuse past generated examples during training. This idea, called experience replay, is widely used in classical reinforcement learning, but has been much less explored for language model post-training. We analyze the tradeoff: older examples may become somewhat outdated, but they can still provide useful signal while avoiding the cost of generating everything from scratch. We find that, when designed carefully, replay buffers can greatly reduce the amount of compute needed during training without hurting final model performance. In some cases, they even improve performance.