Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients

Reinforcement learning from human feedback (RLHF) or verifiable rewards (RLVR), the standard paradigm for aligning LLMs or building recent SOTA reasoning models, is highly sensitive to noise from inconsistent or erroneous rewards. Yet, the interaction between such noise and widely used group-based policy optimization methods remains underexplored. We introduce a noise-robust Group Relative Policy Optimization (GRPO) and Done Right GRPO (Dr.GRPO) framework that explicitly models reward corruption as Bernoulli noise. Our method applies noise correction after estimating reward flip probabilities to debias the learning signal, yielding unbiased gradient estimates. Theoretical analysis shows that group-based methods inherently mitigate individual-level noise, and our correction strategy amplifies this robustness. Empirically, we observe consistent improvements across math and code tasks when applying our noise correction to standard reward model usage, with particular gains of up to 6.7 percentage points in accuracy on math tasks and 1.5 on code tasks under realistic reward model conditions. This work bridges label-noise correction from supervised learning with modern RLHF, offering both theoretical insights and a practical algorithm for noisy real-world deployment.

Large language models are often improved by training them with feedback: good answers receive positive scores and bad answers receive negative scores. For tasks such as math and programming, this feedback is usually produced automatically by a verifier or a reward model. However, these automatic judges are not perfect. They may mark a wrong answer as correct, or reject a correct answer because it is written in an unexpected format. This paper studies how such mistakes affect a popular family of training methods used for reasoning models. We show that noisy feedback does not simply add randomness; it weakens the useful learning signal and can make training converge to a worse model. To address this, we estimate how often the reward source makes each type of mistake and use these estimates to correct the training signal before updating the model. Our experiments show that this correction improves training in both controlled settings, where we artificially add noise, and realistic settings, where feedback comes from existing reward models. On math tasks, the correction improves accuracy by up to 6.7 percentage points, and on code generation by up to 1.5 points. These results suggest that explicitly correcting reward noise can make reinforcement learning for reasoning models more reliable, especially when only a small amount of clean validation data is available.