ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios

Speculative Decodin promises to accelerate Large Language Model inference, yet its efficacy often degrades in production-grade scenarios. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales—particularly the industrial-grade Qwen3-235B—demonstrate that ECHO consistently outperforms state-of-the-art baselines in both low-load and high-load scenarios, achieving up to 5.35$\times$ walltime speedup and delivering over 20\% relative speedup gain against the strongest baselines.

Large language models are powerful, but generating text with them can be slow and expensive because they usually produce one small piece of text at a time. A popular way to speed this up is to let a smaller helper model guess several future pieces of text, while the main model checks which guesses are correct. This works well when the system serves only a few users, but under heavy traffic many guesses are wasted, and checking them can become the main bottleneck. We introduce ECHO, a method that makes this checking process more efficient. Instead of giving every request the same fixed amount of guessing, ECHO estimates which guesses are likely to be useful and reallocates the checking budget within a batch of requests. It stops expanding low-confidence guesses early and spends more computation on requests where extra guesses are more likely to be accepted. We implement ECHO in a production-oriented serving system and evaluate it across several large language models, including very large industrial-scale models. Our results show that ECHO can reduce wasted computation and improve generation speed, especially when many users are served at the same time. This can make large language model services faster and more cost-effective without changing the final verified outputs.