SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Speculative decoding mitigates the memory-bound nature of LLM decoding by using a lightweight draft model to propose multiple tokens for parallel verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source and efficient framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target–draft decoupling, hybrid parallelism, optimized training kernels, and tight integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B compared to the baseline. We further release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs, achieving up to 4.48x end-to-end inference speedup on SGLang and addressing the scarcity of high-quality drafts. Finally, we distill a systematic study of speculative decoding training into practical and actionable recipes to guide real-world adoption.

Large Language Models can be very powerful, but they are also expensive and slow to run because they generate answers one token at a time. This paper introduces SpecForge, an open-source tool designed to make these AI systems respond faster and more efficiently. The main idea is to use a smaller “helper” model to quickly suggest possible next words, while the larger AI model checks those suggestions. When the suggestions are good, the larger model can produce answers much faster without changing the quality of the final response. The authors found that, although this speed-up method is promising, it has not been widely used because good helper models are hard to find and difficult to train. SpecForge addresses this by making the training process faster, more flexible, and easier to use with real-world AI systems. Overall, the work aims to reduce the cost and delay of running large AI models, making advanced AI systems more practical for companies, researchers, and everyday applications.