ConFu: Contemplate the Future for Better Speculative Sampling

Speculative decoding has emerged as a powerful approach to accelerate large language model (LLM) inference by employing lightweight draft models to propose candidate tokens that are subsequently verified by the target model. The effectiveness of this paradigm critically depends on the quality of the draft model. While recent advances such as the EAGLE series achieve state-of-the-art speedup, existing draft models remain limited by error accumulation: they condition only on the current prefix, causing their predictions to drift from the target model over steps. In this work, we propose ConFu (Contemplate the Future), a novel speculative decoding framework that enables draft models to anticipate the future direction of generation. ConFu introduces (i) contemplate tokens and soft prompts that allow the draft model to leverage future-oriented signals from the target model at negligible cost, (ii) a dynamic contemplate token mechanism with MoE to enable context-aware future prediction, and (iii) a training framework with anchor token sampling and future prediction replication that learns robust future prediction. Experiments demonstrate that ConFu improves token acceptance rates and generation speed over EAGLE-3 by 8-11%, across various downstream tasks with Llama-3 3B and 8B models. We believe our work is the first to bridge speculative decoding with continuous reasoning tokens, offering a new direction for accelerating LLM inference.

Large language models are powerful, but generating text with them can be slow because they usually produce one token at a time. A popular way to speed this up is to use a small “draft” model to quickly guess several upcoming tokens, while the large model checks whether those guesses are acceptable. However, the draft model often becomes less accurate as it looks further ahead, because small early mistakes can push it away from what the large model would have written. We propose ConFu, short for “Contemplate the Future,” to help the draft model make better guesses. Instead of asking the draft model to predict only from the current text, ConFu gives it a lightweight hint about the large model’s likely future direction. It does this using special “contemplate” signals that capture the large model’s current intent without requiring expensive extra generation steps. These signals guide the draft model to propose tokens that better match the large model, so more drafted tokens can be accepted at once. Across several language models and tasks, ConFu speeds up generation compared with a strong existing method, making large language models faster to use while preserving their behavior.