Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL

Reinforcement learning with verifiable rewards (RLVR) efficiently scales the reasoning ability of large language models (LLMs) but is bottlenecked by scarce labeled data. Reinforcement learning with intrinsic rewards (RLIR) offers a scalable alternative via self-rewarding, yet often suffers from instability and inferior performance. We trace this gap to a systemic bias in confidence-coupled self-rewarding: the model tends to over-reward high-confidence mistakes, forming a \textbf{self-confirming loop}. We quantify this feedback-loop bias with three metrics: reward noise magnitude ($\rho_{\text{noise}}$), policy–reward coupling ($\rho_{\text{selfbias}}$), and over-/under-reward skew ($\rho_{\text{symbias}}$). Our analyses show a compounding effect where strong coupling amplifies confidence-conditioned errors and drives a drift toward over-reward, leading to instability and a lower performance ceiling. To mitigate this, we propose reinforcement learning with ensembled rewards (\textbf{RLER}), which aggregates diverse models with adaptive reward interpolation and disagreement-aware rollout selection to reduce coupling and suppress over-reward drift. Extensive experiments show that RLER improves by 6.2\% over the best RLIR baseline and is within 3.6\% of RLVR, while exhibiting stable scaling on unlabeled samples.

Large language models can improve their reasoning by training on problems with checkable answers, but such labeled data is often limited. A natural alternative is to let the model learn from its own generated answers. However, this can create a self-confirming loop: when the model is confidently wrong, it may assign overly positive feedback to its own mistake and reinforce it during training. This paper studies this failure mode and shows that it comes from three related issues: noisy self-generated rewards, rewards that are too tied to the model’s own predictions, and an imbalance between over-rewarding wrong answers and under-rewarding correct ones. We propose RLER, which estimates rewards using an ensemble of models, adaptively balances hard and soft reward signals, and selects training examples based on model disagreement. This makes self-rewarded training more stable and improves reasoning performance on unlabeled data.