IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of LLMs remain poorly understood. We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps. We find that the compute-optimal number of parallel rollouts per problem increases predictably with compute budget and then saturates. This trend holds across both easy and hard problems, though driven by different mechanisms: solution sharpening on easy problems and coverage expansion on hard problems. We further show that increasing the number of parallel rollouts mitigates interference across problems, while the number of problems per batch primarily affects training stability and can be chosen within a broad range. Validated across base models and data distributions, our results recast RL scaling laws as prescriptive allocation rules and provide practical guidance for compute-efficient LLM RL post-training.

Large language models can often be improved after their initial training using reinforcement learning, a process where they try problems, receive feedback on whether their answers are correct, and learn from those attempts. This process can be very expensive, and researchers often face a practical question: with a fixed training budget, should a model try many answers to the same problem, see more different problems, or train for more rounds? This paper studies how to make that choice. We run experiments across several language models and problem collections, and we find a clear pattern: as the available budget grows, it is usually better to let the model try more answers for each problem, up to a point where the benefit levels off. For easier problems, this makes correct behavior more reliable. For harder problems, it helps the model discover successful solutions that are otherwise rare. Our results give researchers a practical guide for running reinforcement learning—the process of improving models through repeated attempts and feedback—more efficiently, predictably, and reproducibly.