Esoteric Language Models: A Family of Any-Order Diffusion LLMs

Diffusion Language Models offer a compelling alternative to autoregressive (AR) models by enabling parallel and controllable generation. Within this family, Masked Diffusion Models (MDMs) currently perform best but still underperform AR models in perplexity and lack key inference-time efficiency features, most notably KV caching. We introduce Esoteric Language Models (Eso-LMs), a new family of models that fuses AR and MDM paradigms, smoothly interpolating between their perplexities while overcoming their respective limitations. Unlike prior work, which uses transformers with bidirectional attention as MDM denoisers, we exploit the connection between MDMs and Any-Order autoregressive models and adopt causal attention. This design lets us (1) compute the exact likelihood of MDMs for the first time and, crucially, (2) allows exact KV caching for MDMs while preserving parallel generation over the full sequence length for the first time, significantly improving inference efficiency. Combined with an optimized sampling schedule, Eso-LMs establish a new state of the art on the speed-quality Pareto frontier for unconditional generation.

Today’s most common language models write text one piece at a time from left to right, which works well but can be slow. Diffusion language models offer a different approach: they can fill in many missing pieces of a sentence in parallel, but they are usually less efficient in practice and often produce lower-quality text. This paper introduces Esoteric Language Models, or Eso-LMs, which combine the strengths of both approaches. The model first fills in parts of a sequence in parallel, like a diffusion model, and then completes the remaining gaps one by one, like a standard language model. A key technical change lets Eso-LMs reuse previous computation during generation, making diffusion-style language models much faster without giving up parallel generation. The method also lets researchers smoothly choose between faster, more parallel generation and higher-quality, more sequential generation. In experiments, Eso-LMs achieve a better trade-off between speed and quality than prior diffusion-based language models, especially for long texts. The work also provides a new way to compute exact likelihoods for these models, helping researchers better evaluate and train them. Overall, Eso-LMs are a step toward faster and more flexible language generation systems.