Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement

Large language models (LLMs) have exhibited remarkable performance on complex reasoning tasks, with reinforcement learning under verifiable rewards (RLVR) emerging as a principled framework for aligning model behavior with reasoning chains. Despite its promise, RLVR remains prohibitively resource-intensive, requiring extensive reward signals and incurring substantial rollout costs during training. In this work, we revisit the fundamental question of data and compute efficiency in RLVR. We first establish a theoretical lower bound on the sample complexity required to unlock reasoning capabilities, and empirically validate that strong performance can be achieved with a surprisingly small number of training instances. To tackle the computational burden, we propose Dynamic One-Shot Policy Refinement (DoPR), a uncertainty-aware RL strategy that dynamically selects a single informative training sample per batch for policy updates, guided by reward volatility and exploration-driven acquisition. DoPR reduces rollout overhead by nearly an order of magnitude while preserving competitive reasoning accuracy, offering a scalable and resource-efficient solution for LLM post-training. This approach offers a practical path toward more efficient and accessible RL-based training for reasoning-intensive LLM applications.

Large language models are becoming increasingly good at solving difficult problems. A common way to improve these abilities is to let the model practice many problems until it reaches the correct answer. However, this training process can be extremely expensive, because the model must generate many attempted solutions. In this work, we study how much data and computation are actually needed to improve reasoning in LLMs. We first show, both theoretically and experimentally, that strong reasoning ability can sometimes be activated with far fewer training examples than commonly expected. Based on this insight, we propose Dynamic One-Shot Policy Refinement, a training method that updates the model using only the most informative example in each batch. The method chooses this example by looking for problems where the model’s rewards are uncertain or changing rapidly. Our approach greatly reduces training cost while maintaining competitive reasoning performance. This makes reinforcement-learning-based improvement of reasoning models more practical, scalable, and accessible.