VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier via intra-model routing, instead of the full verifier. This motivates our slim-verifier to handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), a multi-tier framework using a routed slim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases, slim-verifier regeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10–0.22 and delivers 10–20% speedups over strong SD baselines, while achieving 2.5–3× acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/

Large language models can generate high-quality text, but using them is often slow and expensive because a large model must check or produce each new piece of text. A common way to speed this up is speculative decoding, where a smaller model first drafts possible text and a larger model then verifies it. However, existing methods usually make a simple yes-or-no decision: either accept the draft or ask the full large model to recompute it. Our work introduces VIA-SD, a more flexible multi-stage approach. Instead of sending every uncertain draft directly to the full model, VIA-SD adds a slim verifier derived from the large model itself. This slim verifier handles medium-difficulty cases: tokens that are too uncertain to accept immediately, but not difficult enough to require the full model. By reducing unnecessary calls to the largest model, VIA-SD makes text generation faster while keeping output quality stable. It can also be added to existing speculative decoding systems without retraining them, making it a practical step toward more efficient large language model inference.