GRPO is Secretly a Process Reward Model

Process reward models (PRMs) allow for fine-grained credit assignment in reinforcement learning (RL), and seemingly contrast with outcome reward models (ORMs), which assign a single reward to an entire trajectory. However, we provide theoretical proof in this work that the Group Relative Policy Optimization (GRPO) RL algorithm equipped with an ORM is in fact equivalent to a PRM-aware RL objective equipped with a non-trivial, Monte-Carlo-based PRM (given mild assumptions). Leveraging the framework of GRPO-as-a-PRM, we identify a flaw in the GRPO objective that interacts with imbalanced process steps and rewards to hinder both exploration and exploitation (under different conditions). We propose a simple modification to the algorithm to mitigate this defect ($\lambda$-GRPO), and show that LLMs tuned with $\lambda$-GRPO outperform LLMs tuned with standard GRPO on downstream reasoning tasks$\textemdash$and reach peak performance more rapidly. These results show that we can leverage the hidden, built-in PRM structure within the vanilla GRPO algorithm to boost model performance without employing an explicit PRM, and with a negligible impact on training time and cost.

When teaching an AI language model to solve reasoning problems, we guide them with rewards: numerical scores that signal how good the model’s solution is. The simplest approach is to give a single reward based on the final answer, but for multi-step problems, this doesn’t tell the model anything about the value of each individual step. Although scoring each step can be more informative, step-level reward systems are difficult and expensive to train correctly. Our key finding: GRPO—a popular and effective training method that uses only a single final score—is “secretly” already doing step-level scoring. When a model produces several solutions that all start with the same steps, GRPO rewards those shared opening steps based on the kind of final score that they tend to lead to. However, this implicit step-level reward mechanism over-weights frequently appearing steps, which can imbalance training. We propose a simple re-balancing method, $\lambda$-GRPO, and demonstrate that models trained with $\lambda$-GRPO learn faster and solve reasoning problems more accurately. This shows that the benefits of step-level reward can be unlocked from a training method already in wide use, without actually building a costly, step-level reward system.