SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs

Recent studies observe that reinforcement learning with verifiable rewards (RLVR) reliably improves pass@1 on reasoning tasks, yet often fails to yield comparable gains in pass@k, raising the question of whether RLVR genuinely enables large language models to acquire novel reasoning abilities or merely enhances the efficiency of sampling reasoning modes already present in the base model. Prior analyses largely support the latter view, attributing this limitation to structural properties of standard RLVR objectives that result in insufficient exploration pressure. In this work, we argue that a central structural constraint arises from reverse-KL regularization, which stabilizes training but inherently anchors the policy to the reference distribution, thereby suppressing the emergence of alternative reasoning modes. However, we show that neither removing the KL term nor replacing it with forward-KL provides a satisfactory solution, as both disrupt the efficiency–coverage trade-off by either inducing reward hacking or allocating probability mass to off-target regions. To resolve this tension, we propose SAGE, a principled framework that enables controllable empirical support expansion by reshaping the reverse-KL anchor distribution itself through a guide function $q(x,y)$, achieving consistent improvements in both pass@1 and pass@k across challenging mathematical reasoning benchmarks.

When AI models learn to solve math problems through trial and error, they gradually become biased toward the reasoning strategies they already use most often-much like a student who repeatedly relies on a familiar method because it has worked before. As training progresses, these dominant strategies become increasingly reinforced, while other valid but less common approaches are explored less frequently. One reason is a stabilization mechanism called KL regularization, which keeps the model close to its starting behavior but can unintentionally concentrate learning on already dominant reasoning modes. Rather than removing this stabilizing anchor-which can cause models to exploit flaws in the reward signal-we redesign what the model is anchored to. Our method, SAGE, adaptively reshapes the anchor using simple internal signals, such as the model’s uncertainty during reasoning. At points where multiple solution paths appear plausible, SAGE gently encourages exploration of underrepresented reasoning strategies while preserving the stability benefits that make KL regularization effective. Importantly, SAGE does not require a new training algorithm. Instead, it acts as a lightweight, plug-and-play enhancement that can be seamlessly integrated into existing reinforcement learning methods. This makes it easy to improve exploration without increasing training complexity or sacrificing stability. Models trained with SAGE solve more math problems correctly, discover a broader range of valid reasoning strategies, and generalize better to new problems. By encouraging exploration while retaining the safeguards that keep training stable, SAGE helps AI systems learn more effectively from their own experience.