EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

Reinforcement learning (RL) has become a representative post-training paradigm for large language models (LLMs), enabling strong reasoning and agentic capabilities. However, its rollout generation remains a dominant training bottleneck because it relies on sequential autoregressive (AR) decoding, where a small number of long-tailed responses often determine completion time. Speculative decoding (SD) can reduce inference latency while preserving model quality by drafting tokens and verifying them in parallel. Applying SD to RL rollouts, however, introduces challenges absent from standard LLM inference: (i) algorithmically, the continuously evolving target model makes static drafters stale; (ii) system-wise, rollout decoding moves across regimes, from large active batches where SD can be compute-bound and ineffective to shrinking-batch tails where SD becomes beneficial. We present EfficientRollout, an SD framework designed to accelerate RL rollouts by addressing these RL-specific challenges. EfficientRollout induces a quantized drafter directly from the target model, keeping it coupled to the evolving policy without separate drafter training. It enables SD only in roofline-predicted beneficial regimes and adapts drafting budgets using training-time acceptance signals. Under a realistic RL workload, EfficientRollout reduces rollout and end-to-end latency by up to 19.6% and 12.7% over a standard accelerated AR rollout baseline while preserving training dynamics.