Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

Reinforcement Learning with Verifiable Reward (RLVR) is a powerful method for enhancing the reasoning abilities of Large Language Models, but its full potential is limited by a lack of exploration in two key areas: \textbf{Depth} (the difficulty of problems) and \textbf{Breadth} (the number of training instances). Our analysis of the popular GRPO algorithm reveals a bias that down-weights difficult, low-accuracy problems, which are crucial for improving reasoning skills. To address this, we introduce Difficulty Adaptive Rollout Sampling (DARS), a method that re-weights difficult problems by using targeted, multi-stage rollouts. This approach increases the number of rollout outcomes for these harder problems according to our proposed re-balancing schedules and leads to consistent gains in \textit{Pass@K}. We also found that simply enlarging the rollout size isn't effective and can even harm performance. We also investigated the role of breadth by scaling the batch size and using full-batch updates. This significantly improved \textit{Pass@1} performance by maintaining high token-level entropy, which indicates continued exploration and reduced gradient noise. Finally, we present DARS-Breadth, a combined approach that uses DARS with a large breadth of training data. This method demonstrates simultaneous gains in both \textit{Pass@K} and \textit{Pass@1}, confirming that depth (adaptive exploration) and breadth (scaling the training data) are orthogonal and essential dimensions for unlocking the full reasoning power of RLVR.

Large language models (LLMs) have made remarkable progress in solving complex reasoning tasks like mathematics and programming in recent years. This breakthrough is mainly driven by a powerful training method called Reinforcement Learning with Verifiable Rewards (RLVR), which can automatically check whether a model's answer is correct without manual labeling, enabling efficient large-scale training. However, we found two critical flaws that limit the effectiveness of current RLVR systems. First, the most popular RLVR algorithm naturally focuses on medium-difficulty problems and largely overlooks the hardest ones—exactly the problems that help models push their reasoning limits. Second, simply giving the model more attempts per problem does not improve performance and can even backfire, while the benefit of training on more problems in each step has been long underestimated. To address these issues, we developed Difficulty Adaptive Rollout Sampling (DARS). This method first quickly assesses the difficulty of each problem, then intelligently allocates extra attempts only to the hardest problems, allowing the model to learn valuable reasoning skills from these challenging examples. We also discovered that training on many more problems at once makes the model more stable and significantly improves its accuracy on the first try. By combining DARS with this large-scale training approach (DARS-Breadth), we achieved simultaneous gains in both how often the model answers correctly on the first attempt and how well it can solve extremely hard problems when given multiple tries. Our work proves that practicing harder problems and practicing more problems are complementary, not competing, and together they unlock the full reasoning potential of large language models.