SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optimizations, SD performance is inherently data-dependent, meaning that diverse and representative workloads are essential for accurately measuring its effectiveness. Existing benchmarks suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments. To address this, we introduce SPEED-Bench, a comprehensive suite designed to standardize SD evaluation across diverse semantic domains and realistic serving regimes. SPEED-Bench offers a carefully curated Qualitative data split, selected by prioritizing semantic diversity across the data samples. Additionally, it includes a Throughput data split, allowing speedup evaluation across a range of concurrencies, from latency-sensitive low-batch settings to throughput-oriented high-load scenarios. By integrating with production engines like vLLM and TensorRT-LLM, SPEED-Bench allows practitioners to analyze system behaviors often masked by other benchmarks. We highlight this by quantifying how synthetic inputs overestimate real-world throughput, identifying batch-size dependent optimal draft lengths and biases in low-diversity data, and analyzing the caveats of vocabulary pruning in state-of-the-art drafters. We release SPEED-Bench to establish a unified evaluation standard for practical comparisons of SD algorithms.

Large language models are increasingly used for a wide range of applications, but generating their responses can be slow. Speculative decoding is an important acceleration technique that improves inference speed by proposing candidate tokens with a lightweight draft model and then verifying them with the target model. However, its effectiveness depends heavily on the input data, the task, and the serving conditions. As a result, benchmarks that use narrow, synthetic, or low-diversity workloads can give an inaccurate picture of real-world performance. We introduce SPEED-Bench, a benchmark for evaluating speculative decoding under more realistic and diverse conditions. SPEED-Bench includes a qualitative split designed to cover varied semantic domains, as well as a throughput split that measures speedups across different concurrency levels, from latency-sensitive settings to high-load serving scenarios. It also integrates with production inference engines such as vLLM and TensorRT-LLM. Using SPEED-Bench, we show that synthetic inputs can overestimate throughput, that optimal draft lengths depend on batch size, and that low-diversity data can bias evaluation results. SPEED-Bench provides a shared standard for comparing speculative decoding methods in settings that better reflect practical deployment.