IDLM: Inverse-distilled Diffusion Language Models

Diffusion Language Models (DLMs) have recently achieved strong results in text generation. However, their multi-step sampling leads to slow inference, limiting practical use. To address this, we extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting. Nonetheless, this extension introduces both theoretical and practical challenges. From a theoretical perspective, the inverse distillation objective lacks uniqueness guarantees, which may lead to suboptimal solutions. From a practical standpoint, backpropagation in the discrete space is non-trivial and often unstable. To overcome these challenges, we first provide a theoretical result demonstrating that our inverse formulation admits a unique solution, thereby ensuring valid optimization. We then introduce gradient-stable relaxations to support effective training. As a result, experiments on multiple DLMs show that our method, *Inverse-distilled Diffusion Language Models (IDLM)*, reduces the number of inference steps by $4 \times$-$64 \times$, while preserving the teacher model’s generation quality. We provide the code, model checkpoints, and video tutorials on the project page: [https://david-cripto.com/idlm](https://david-cripto.github.io/idlm-project-page)

Diffusion language models generate text by gradually refining an initially noisy sequence, but this process can require hundreds or thousands of steps, making generation slow. We introduce IDLM, a training method that teaches a smaller sampling process to imitate a pretrained diffusion language model in far fewer steps. Instead of training a new language model from scratch, IDLM uses an existing model as a teacher and learns how to preserve its generation behavior while shortening the sampling procedure. Our experiments show that IDLM can substantially reduce the number of generation steps across several diffusion language models while maintaining strong text quality. This suggests a practical way to make diffusion-based text generation faster and more usable.