Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen2.5 achieve significant gains even with spurious rewards. We investigate this phenomenon and identify ``Perplexity Paradox'': spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting model is bypassing reasoning in favor of memorization. Using a suite of mechanistic interpretability tools, including Path Patching and Logit Lens, we identify a previously unknown Anchor–Adapter circuit. This circuit enables model to bypass reasoning and directly retrieve memorized solutions under spurious RLVR. We localize a Functional Anchor in middle layers (L18–20) that triggers retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering, i.e., artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models.

Recent studies found that some AI models can get higher math scores even when they are trained with unreliable rewards, such as random or wrong feedback. This suggests that the models may not always be learning better problem-solving skills. Instead, they may sometimes be recalling answers from problems they have already seen. Our paper studies how this happens inside the model. We find that unreliable training can activate a shortcut that helps the model retrieve memorized answers. This shortcut appears in a specific part of the model and can be weakened or strengthened by targeted interventions. When we weaken it, suspicious gains on previously seen questions decrease, while performance on fresh questions is mostly unchanged. These results help explain why some benchmark improvements can be misleading and provide a way to check whether a model is really reasoning or relying on memory.