XRPO: Pushing the Limits of GRPO with Targeted Exploration and Exploitation

Reinforcement learning algorithms such as GRPO have driven recent advances in large language model (LLM) reasoning. While scaling the number of rollouts stabilizes training, existing approaches suffer from limited exploration on challenging prompts and leave informative feedback signals underexploited, due to context-independent rollout allocation across prompts (e.g., generating 16 rollouts per prompt) and relying heavily on sparse rewards. This paper presents XRPO (eXplore–eXploit GRPO), a unified framework that recasts policy optimization through the principled lens of rollout exploration–exploitation. To enhance exploration, XRPO introduces a mathematically grounded rollout allocator that adaptively prioritizes prompts with higher potential for uncertainty reduction. It further addresses stagnation on zero-reward prompts through an in-context seeding strategy that injects curated exemplars, steering the model into more difficult reasoning trajectories. To strengthen exploitation, XRPO develops a group-relative, novelty-aware advantage sharpening mechanism that leverages sequence likelihoods to amplify low-probability yet correct responses, thereby extending the policy’s reach beyond sparse rewards. Experiments across diverse math and coding benchmarks on both reasoning and non-reasoning models demonstrate that XRPO outperforms existing advances (e.g., GRPO and GSPO) up to 4% pass@1 and 6% cons@32, while accelerating training convergence by up to 2.7x.

When AI models learn to reason through math or code, they practice by generating many candidate answers and learning from correctness feedback. Current methods waste effort by treating all problems equally and ignoring differences among correct answers, leaving the hardest problems unsolved and diverse reasoning strategies unexplored. We introduce XRPO, a training framework that intelligently focuses practice on problems where the model is most uncertain, seeds unsolved problems with worked examples to break through capability limits, and amplifies learning from correct answers that use uncommon reasoning paths. Across challenging benchmarks, XRPO improves accuracy by up to 4% and converges 2.7 times faster than existing methods.