Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing

Large Language Models (LLMs) possess latent multi-token prediction (MTP) abilities despite being trained only for next-token generation. We introduce ESP (Embedding-Space Probing), a simple and training-free MTP method that probes an LLM using on-the-fly mask tokens drawn from its embedding space, enabling parallel future-token prediction without modifying weights or relying on draft models. ESP constructs a speculative token tree by sampling Top-K candidates from mask-token logits and applies a lightweight pruning rule to retain high-probability continuations. During generation, predictions are verified in parallel, yielding lossless decoding while significantly reducing model calls and increasing token throughput. ESP consistently outperforms existing training-free baselines, improving acceptance length by $7$–$11$% over Lookahead Decoding on LLaMA3 and $7$–$8$% on Qwen3, and increasing throughput by up to $15$–$19$% over the strongest baseline. Finally, we provide theoretical insight and empirical evidence showing that decoder layers naturally align mask-token representations with next-token states, enabling accurate multi-step prediction without retraining or auxiliary models.

Modern AI systems like large language models (LLMs) are typically designed to generate text one word at a time. However, this step-by-step process can be slow, especially for long outputs. Interestingly, these models often have an untapped ability to predict multiple future words at once—but existing methods to use this capability usually require retraining the model or adding extra components. In this work, we introduce ESP (Embedding-Space Probing), a simple and training-free method to unlock this hidden ability. Instead of modifying the model or training new components, ESP inserts temporary “mask tokens” into the model’s input in a way that prompts it to predict several future words simultaneously. These predictions are then verified to ensure correctness, allowing faster text generation without sacrificing accuracy. To organize these predictions efficiently, ESP builds a tree of possible future words and prunes low-quality options. This allows the method to focus on the most promising continuations while keeping computation manageable. Importantly, everything is done using the original, unchanged model. We evaluate ESP across multiple state-of-the-art language models and tasks. ESP consistently speeds up text generation—reducing the number of model calls and improving throughput—while maintaining exact output quality. It outperforms existing training-free methods by a significant margin. Overall, our results show that current language models already possess strong multi-step prediction capabilities. ESP provides a simple way to harness this ability, enabling faster and more efficient AI systems without additional training or infrastructure.