Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

Diffusion language models (dLMs) have emerged as a promising paradigm enabling parallel generation, but their learning efficiency lags behind that of autoregressive (AR) language models when trained from scratch. To this end, we study AR-to-dLM conversion, which transforms pretrained AR models into efficient dLMs that excel in speed while preserving AR models’ task accuracy. We achieve this by identifying limitations in the attention patterns and objectives of existing AR-to-dLM methods and then proposing methodologies and actionable insights for scalable AR-to-dLM conversion. Specifically, we first systematically compare different attention patterns and find that maintaining pretrained AR weight distributions is key to effective AR-to-dLM conversion. Accordingly, we introduce a continuous pretraining scheme with a block-wise attention pattern. We find that, in addition to block-wise attention’s known benefit of enabling KV caching, its block-wise causality better preserves pretrained AR models’ weight distributions, leading to a win–win in accuracy and efficiency. Second, to mitigate the training–test gap in mask token distributions (uniform vs. highly left-to-right), we propose a position-dependent token masking strategy that assigns higher masking probabilities to later tokens during training to better mimic test-time behavior. These studies lead to the Efficient-DLM model family, which outperforms state-of-the-art AR models and dLMs in accuracy–throughput trade-offs; for example, our Efficient-DLM-8B achieves +5.4\%/+2.7\% higher accuracy with 4.7$\times$/2.8$\times$ higher throughput compared to Dream-7B and Qwen3-4B, respectively.

Most language models generate text one word at a time, which can make them slow, especially when serving a small number of users. An alternative approach can generate several words in parallel, but existing models of this kind are often harder to train and have not consistently matched the quality of standard language models. In this paper, we study how to turn strong existing language models into faster parallel-generation models without losing their original abilities. The key idea is to retrain the model in a way that keeps its useful left-to-right knowledge while letting it fill in small chunks of text more flexibly. We also adjust training so that it better matches how the model generates text at test time. These design choices lead to Efficient-DLM, a family of models that improves the trade-off between answer quality and generation speed. Across math, coding, knowledge, and reasoning tasks, Efficient-DLM produces competitive or better answers while generating text substantially faster than prior models.