Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a central paradigm for scaling LLM reasoning, yet its optimization often suffers from training instability and suboptimal convergence. Through a systematic dissection of clipping-based GRPO-style objectives, we identify the rigid clipping decision induced by hard clipping as a key practical bottleneck in the studied RLVR setups. Specifically, our analysis suggests that informative signals can lie in the near-boundary region just beyond the clipping threshold, and are therefore discarded by the standard hard-clipping rule. Motivated by this diagnosis, we propose Near-boundary Stochastic Rescue (NSR), a minimal, plug-and-play modification that stochastically retains these slightly out-of-bound tokens to recover lost signals. While NSR, via stochastic sampling, can be interpreted as inducing an implicit gradient decay in expectation, our ablations reveal that its stochastic, boundary-local rescue mechanism is consistently more effective than deterministic gradient decay. Validated by extensive experiments across model sizes from 7B to 30B and both dense and MoE architectures, as a plug-and-play solution, NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO. Our code is publicly available at https://github.com/qwenpilot/NSR.

Large language models are increasingly trained to solve reasoning problems, such as math and coding tasks, by trying possible answers and receiving rewards when an answer can be automatically checked as correct. However, this training process can be unstable: a model may stop improving, change too abruptly, or lose useful behaviors during training. This paper studies one cause of this instability. Many current training methods use a strict safety rule that blocks updates when they appear to move the model too far from its previous behavior. We find that this rule can be too rigid: it may throw away useful learning signals that are only slightly outside the allowed range. To address this, we introduce Near-boundary Stochastic Rescue, a simple modification that sometimes keeps these slightly discarded signals instead of always removing them. The method does not broadly relax the safety rule; it only gives a small chance to recover signals close to the boundary. Across several large language models and reasoning benchmarks, this approach makes training more stable and often improves final performance. Its simplicity also makes it easy to add to existing training pipelines.