When RL Meets Adaptive Speculative Training: A Unified Training-Serving System

Speculative decoding can significantly accelerate LLM serving, yet most deployments today disentangle speculator training from serving, treating speculator training as a standalone offline modeling problem. We show that this decoupled formulation introduces substantial deployment and adaptation lag: (1) high time-to-serve, since a speculator must be trained offline for a considerable period before deployment; (2) delayed utility feedback, since the true end-to-end decoding speedup is only known after training and cannot be inferred reliably from acceptance rate alone due to model-architecture, diverse prompt engineering, and system-level overheads; and (3) domain-drift degradation, as the target model is repurposed to new domains and the speculator becomes stale and less effective. To address these issues, we present Aurora, a unified training–serving system that closes the loop by continuously learning a speculator directly from live inference traces. Aurora reframes online speculator learning as an asynchronous reinforcement-learning problem: accepted tokens provide positive feedback, while rejected speculator proposals provide implicit negative feedback that inherits the online traffic failure signals. Our design integrates an SGLang-based inference server with an asynchronous training server, enabling hot-swapped speculator updates without service interruption. Crucially, Aurora supports day-0 deployment: a speculator can be served immediately and rapidly adapted to live traffic, improving system performance while providing immediate utility feedback. Across experiments, Aurora achieves a 1.5$\times$ day-0 speedup on recently released frontier models (e.g., MiniMax M2.1 229B and Qwen3-Coder-Next 80B). Aurora also adapts effectively to distribution shifts in user traffic, delivering an additional 1.25$\times$ speedup over a well-trained but static speculator on widely used models (e.g., Qwen3 and Llama3).

Large AI language models are powerful but slow and expensive to run, because they generate text one word at a time. A popular speed-up trick pairs the big model with a small, fast "helper" that guesses several words ahead, which the big model then quickly checks and accepts—skipping a lot of slow work. The catch: this helper is usually trained once, in advance, and gradually becomes outdated as the kinds of questions real users ask change, so its speed benefit fades. It also requires a long, costly training phase before it helps at all. We built Aurora, a system that trains the helper continuously while the model is actively serving real users. It learns on the fly from which guesses succeed and which fail, and seamlessly swaps in an improved helper without ever interrupting service. Aurora can even start from scratch on the first day a new model is deployed, and quickly becomes useful. In practice, Aurora makes serving today's largest open models about 1.5 times faster and keeps adapting as user demand shifts, lowering the cost, energy, and waiting time of using AI assistants.