MineDraft: A Framework for Batch Parallel Speculative Decoding

Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of MineDraft in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems. The code is publicly available in the MineDraft GitHub repository.

Large language models like GPT or Llama are slow to generate text because they produce one word at a time. A popular trick to speed this up is called speculative decoding: a small, fast "draft" model guesses several upcoming words, and then the big, accurate model checks those guesses all at once. This avoids making the big model do all the work from scratch. The catch with the standard approach is that the drafting and checking steps happen one after the other, first draft, then verify, then draft again. This "wait your turn" setup wastes time, because the big model sits idle while the small model is drafting, and vice versa. MineDraft fixes this with a smarter batch parallel system. Instead of one queue of requests, MineDraft keeps two queues running simultaneously: while the big model is checking drafts for one batch of users, the small model is already drafting for the next batch. The two stages overlap, so neither model is ever sitting idle waiting for the other. MineDraft delivers up to 75% more throughput (serving more users in the same time) and up to 39% lower latency (faster response times for individual users) compared to standard speculative decoding. The authors also built it as a plug-in for vLLM, a widely-used production inference engine, showing it can be deployed in real-world systems without a full rewrite.