Reinforcement Learning with Verifiable Rewards: GRPO's Loss, Dynamics, and Success Amplification

Group Relative Policy Optimization (GRPO) was introduced recently and used to train DeepSeek\textendash R1 for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean{+}variance calibration of these rewards induces a contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs.\ mean{+}variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (\emph{mirror}), penalizing divergence from a fixed reference model $\pi_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $\pi_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $\pi_{n-1}$, as well as the policies $\pi_{n-1}$ and $\pi_{\mathrm{ref}}$. Iterating results in a sequence $\{\pi_n\}$ whose \emph{probability of success (PoS)} obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy's probability of success.

Modern AI systems can improve by practicing: they generate answers, receive feedback on whether those answers are correct, and update themselves from that feedback. A popular method for this kind of training is GRPO, which has been used to improve reasoning in large language models. However, it is not fully clear why GRPO works, when it improves a model, or when it may fail. This paper studies GRPO in the important setting where feedback is simply right or wrong, such as solving a math problem correctly or passing a code test. We show how different ways of processing this feedback change what the model learns and how likely it is to give correct answers over time. We compare several versions of GRPO, including ones that stay close to the original model and ones that mainly follow the most recent model. Our results explain when training should steadily improve success, when it may stop improving, and how noisy feedback can hurt learning. Experiments on math questions support these predictions