Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptance

Speculative decoding accelerates inference for (M)LLMs, yet a training-decoding discrepancy persists: while existing methods optimize single greedy trajectories, decoding involves verifying and ranking multiple sampled draft paths. We propose Variational Speculative Decoding (VSD), formulating draft training as variational inference over latent proposals (draft paths). VSD maximizes the marginal probability of target-model acceptance, yielding an ELBO that promotes high-quality latent proposals while minimizing divergence from the target distribution. To enhance quality and reduce variance, we incorporate a path-level utility and optimize via an Expectation-Maximization procedure. The E-step draws MCMC samples from an oracle-filtered posterior, while the M-step maximizes weighted likelihood using Adaptive Rejection Weighting (ARW) and Confidence-Aware Regularization (CAR). Theoretical analysis confirms that VSD increases expected acceptance length and speedup. Extensive experiments across LLMs and MLLMs show that VSD achieves up to a 9.58\% speedup over EAGLE-3 and 8.80\% over ViSpec, significantly improving decoding efficiency.

Speculative decoding accelerates large language model inference by allowing a lightweight draft model to propose multiple tokens that are then verified in parallel by the target model. However, existing draft models are typically trained to optimize a single greedy trajectory at the token level, whereas practical decoding evaluates and ranks multiple candidate paths at the sequence level. This training-decoding mismatch can degrading the effectiveness of draft model training and limit attainable decoding speedups. To address this issue, we introduce Variational Speculative Decoding (VSD), a principled framework that reformulates draft-model training as variational inference over latent draft paths. Rather than merely matching token-wise target likelihoods, VSD encourages the draft model to generate path-level proposals that are more likely to be accepted by the target verifier. To optimize this path-level objective, we further develop a Monte Carlo EM procedure that improves training stability and enables the model to effectively distinguish informative proposals from invalid ones. Across both language and multimodal models, VSD improves acceptance length and inference efficiency, achieving up to 9.6\% speedup over EAGLE-3 and 7.9\% over ViSpec. These results show that aligning draft model training with the actual sequence-level behavior of speculative decoding is important for building faster and more efficient generative systems.