DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning

Reinforcement Learning (RL) has become pivotal for improving model capabilities yet suffers from rollout efficiency bottlenecks due to the long-tail response length distribution. While existing works mitigate the impact of long tails via prompt-level tail scheduling, we focus on the root source of inefficiency: the distribution itself. Specifically, we characterize the long-tail distribution at a finer granularity, identifying intra-prompt long tails, and revealing that they frequently consist of ineffective verbosity. To address this, we propose a novel paradigm of active distribution shaping to shape the rollout distribution towards conciseness and certainty, thereby fundamentally resolving tail-induced overheads. We achieve this through a distribution-aware trajectory sampling mechanism, which selects trajectories from a redundant exploration space for each prompt, and an adaptive redundancy allocation scheme to maximize both shaping effectiveness and system efficiency. Experiments demonstrate significant acceleration over state-of-the-art systems by up to 1.77$\times$ without compromising model performance.

Training large language models to reason through reinforcement learning is powerful, but it is also expensive and slow. During training, the model repeatedly tries to answer questions and learns from the results. A major bottleneck is that some answers become extremely long: they consume a lot of computation, and in many training systems the whole batch must wait for the longest answer to finish. In this work, we show that this problem is not only caused by some questions being harder than others. Even for the same question, the model may sometimes produce concise answers and sometimes produce unnecessarily long ones. We propose DARTS, a training framework that actively reduces this wasted verbosity. DARTS generates multiple candidate answers for each question, keeps mostly concise ones for learning, and still preserves a small number of long answers when deeper reasoning may be useful. By shaping the model’s training data in this way, DARTS makes reinforcement learning more efficient without simply forcing the model to be short. In our experiments, DARTS speeds up large language model reinforcement learning by up to 1.77× while maintaining model accuracy. This can make advanced reasoning models cheaper, faster, and more practical to train.