NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies

The massive vocabulary sizes of large language models, often exceeding 100k tokens, impose a computational bottleneck on the final linear projection layer during speculative decoding. Existing vocabulary pruning solutions rely on static or coarsely-grained sub-vocabularies that necessitate large active sizes ($\sim$30k) to maintain draft quality. We propose NanoSpec, a novel training-free approach that breaks this trade-off by dynamically constructing a minimalist, context-aware active vocabulary for each generation step. Leveraging the inherent temporal locality of language generation, NanoSpec achieves high coverage while slashing the average vocabulary size by over $40\times$ (to $<$3k tokens) without requiring any auxiliary trained parameters. To realize the theoretical benefits of such high sparsity on modern hardware, we introduce a system-algorithm co-design that overcomes the inefficiencies of sparse memory access through asynchronous gathering and GPU-resident state management. As a complementary plug-and-play module, NanoSpec cuts draft time by an average of 51.6\%, delivering a $1.17$-$1.29\times$ end-to-end speedup over the state-of-the-art speculative decoding methods EAGLE-2 and EAGLE-3 across 7 tasks and outperforming complex training-based pruning baselines.

Large language models generate text one word at a time, which makes them slow. A popular trick called "speculative decoding" uses a small, fast helper model to guess multiple words ahead, then checks them all at once. However, this helper still wastes time searching through massive vocabularies of over 100,000 possible words. We built NanoSpec, a system that shrinks the helper model's search space from 100,000+ words down to fewer than 3,000 at each step, by observing that the next word almost always relates to what was just written. Instead of relying on fixed word lists or trained classifiers, NanoSpec dynamically picks the most relevant words from the recent context, where no extra training is required. We also designed GPU-optimized routines so this dynamic selection actually translates into real speed gains rather than being bottlenecked by memory access. NanoSpec plugs into existing speculative decoding systems and speeds them up by 17–29% across diverse tasks including translation, math, coding and conversation, making AI assistants respond noticeably faster without sacrificing quality.