Spurious Rewards: Rethinking Training Signals in RLVR

We show that reinforcement learning with verifiable rewards (RLVR) can elicit strong mathematical reasoning in certain language models even with spurious rewards that have little, no, or outright negative correlation with the correct answer. For example, RLVR training with GRPO improves MATH-500 performance for Qwen2.5-Math-7B in absolute points by 21.4% using randomly assigned rewards, nearly matching the 29.1% gained with ground truth rewards. To explain this counterintuitive observation, we show that GRPO exhibits a clipping bias arising from the clip term, which can amplify high-prior behaviors learned during pre-training even without informative rewards. As a case study, we identify one such high-prior behavior for Qwen2.5-Math models, which we term code reasoning---reasoning in code without actual code execution; code reasoning frequency increases from 65% to over 90% with spurious rewards. However, the presence of such amplifiable behaviors is highly model-dependent. In practice, spurious rewards that are effective for Qwen models often fail to produce gains for other model families, such as Llama3 or OLMo2. Our results highlight the importance of validating RL methods across diverse models rather than relying on a single de facto choice: large performance gains can arise on Qwen models even from random rewards that do not reflect genuine capability improvements.

Reinforcement learning is often used to improve the reasoning ability of language models by rewarding them when they produce correct answers. In math reasoning, these rewards are usually considered reliable because the final answer can be checked automatically. However, this paper shows a surprising result: for some models, reinforcement learning can still appear to improve math performance even when the rewards are random, incorrect, or otherwise unrelated to the true answer. We study why this happens and find that the effect depends strongly on the model being trained. In particular, Qwen2.5-Math models can gain large improvements even from random rewards, while other model families such as Llama and OLMo do not show the same behavior. We explain this through a bias in the GRPO training algorithm, which can amplify behaviors that the model already strongly prefers from pretraining, even when the reward signal itself is not meaningful. One example of such a behavior is what we call code reasoning: writing math solutions in a Python-like style without actually executing code. Qwen2.5-Math models already use this pattern frequently before reinforcement learning, and spurious rewards make it even more common. Because this pattern is correlated with better math performance for these models, amplifying it can improve benchmark scores even without teaching the model new reasoning skills. Overall, our results suggest that large gains from reinforcement learning should be interpreted carefully. Improvements on a single model family may reflect the amplification of existing model-specific behaviors rather than a generally effective training method. We recommend that future work evaluate reinforcement learning methods across diverse model families and include spurious-reward baselines as diagnostic controls.