Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency. Existing compression methods fail in this regime: static eviction incurs accuracy loss due to saliency shift, while dynamic selection introduces prohibitive computational overhead during the verification path. We propose Dustin, a sparse verification framework designed for long-context speculative decoding. Dustin integrates lookahead signals from the draft model with historical attention from the target model to identify critical tokens with high fidelity across multi-step verification windows. To reduce recomputation latency, this approach further employs a sparse estimation scheme that restricts importance scoring to a minimal subset of attention heads. Evaluations on PG-19 and LongBench with Qwen2.5-72B demonstrate that Dustin achieves a 27.85× speedup in self-attention and a 9.17× end-to-end decoding speedup at a 32k sequence length, all with negligible accuracy degradation.

AI language models can now read very long texts — entire books or lengthy reports — but writing a response becomes painfully slow as the text grows. For every new word it produces, the model must sift back through a huge memory of everything it has already read, and fetching that memory is the slowest part of the job. A popular speed-up has a small, fast model guess several words ahead while a large, accurate model checks all the guesses at once — yet this checking step still has to consult the entire memory, so it remains the main bottleneck. We developed Dustin, which lets the large model verify those guesses by examining only the small slice of its memory that truly matters, instead of all of it. The hard part is reliably spotting that slice, because which parts matter keeps shifting as the text is generated. Dustin blends two clues — what the large model focused on most recently, and what the small guessing model expects to need next — and gathers them using only a tiny fraction of the model's internal machinery, so the added cost is negligible. On long documents, Dustin speeds up this bottleneck by up to 28 times and overall text generation by up to 9 times, with virtually no loss in answer quality — making long-context AI far cheaper and faster to run.