Anchored Policy Optimization: Mitigating Exploration Collapse via Support-Constrained Rectification

Reinforcement Learning with Verifiable Rewards (RLVR) is increasingly viewed as a tree pruning mechanism. However, we identify a systemic pathology termed Recursive Space Contraction (RSC), an irreversible collapse driven by the combined dynamics of positive sharpening and negative squeezing, where the sampling probability of valid alternatives vanishes. While Kullback-Leibler (KL) regularization aims to mitigate this, it imposes a rigid Shape Matching constraint that forces the policy to mimic the reference model's full density, creating a gradient conflict with the sharpening required for correctness. We propose Anchored Policy Optimization (APO), shifting the paradigm from global Shape Matching to Support Coverage. By defining a Safe Manifold based on the reference model's high-confidence support, APO permits aggressive sharpening for efficiency while selectively invoking a restorative force during error correction to prevent collapse. We theoretically derive that APO serves as a gradient-aligned mechanism to maximize support coverage, enabling an Elastic Recovery that re-inflates valid branches. Empirical evaluations on mathematical benchmarks demonstrate that APO breaks the accuracy-diversity trade-off, significantly improving Pass@1 while restoring the Pass@K diversity typically lost by standard policy gradient methods.

When we train AI models to solve math problems using reinforcement learning — rewarding correct reasoning and penalizing mistakes — we identify a systemic failure mode we call Recursive Space Contraction (RSC). Each training update makes the model progressively more narrow-minded: correct answers cause it to fixate on one strategy, while penalizing mistakes paradoxically strengthens whatever approach is already dominant rather than encouraging alternatives. RSC creates a self-reinforcing loop that directly causes the model's diversity to collapse — once a valid problem-solving path is suppressed, it becomes mathematically unreachable and can never be recovered. To break RSC, we propose Anchored Policy Optimization (APO). When the model makes an error, instead of blindly suppressing the wrong answer, APO actively restores a set of promising alternatives drawn from the original pre-trained model's knowledge. This gives the model an elastic "safety net" to fall back on when it fails, while still allowing it to sharpen its skills on correct solutions. Experiments on challenging math benchmarks show that APO breaks the usual trade-off between accuracy and diversity — achieving up to 6% higher accuracy while simultaneously preserving the model's ability to explore multiple valid reasoning paths that standard methods permanently destroy.