DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders

Sparse autoencoders (SAEs) have become a standard tool for mechanistic interpretability in autoregressive large language models (LLMs), enabling researchers to extract sparse, human-interpretable features and intervene on model behavior. Recently, as diffusion language models (DLMs) have become an increasingly powerful and promising alternative to autoregressive LLMs, it is essential to develop tailored mechanistic interpretability tools for this emerging class of models. In this work, we present DLM-Scope, the first SAE-based interpretability framework for DLMs, and demonstrate that trained Top-K SAEs can faithfully extract sparse, interpretable features. Notably, we find that inserting SAEs affects DLMs differently from autoregressive LLMs: while SAE insertion in LLMs typically incurs a loss penalty, in DLMs it can reduce cross-entropy loss when applied to early layers, a phenomenon absent or markedly weaker in LLMs. Additionally, SAE features in DLMs enable more effective diffusion-time interventions, often outperforming LLM steering. Moreover, we pioneer new SAE-based research directions for DLMs: we show that SAEs provide useful signals for DLM decoding order, and that SAE features remain stable during DLM post-training. Overall, our work establishes a foundation for mechanistic interpretability in DLMs and highlights the potential of applying SAEs to DLM-related tasks and algorithms.

Modern language models are powerful, but it is still difficult to understand why they produce particular answers. This is especially true for diffusion language models, a newer type of model that writes text by repeatedly refining a partly hidden sentence instead of generating words one by one. We study whether sparse autoencoders—tools that turn a model’s hidden activity into a small set of easier-to-read concepts—can help open up these models. Our system, DLM-Scope, trains these tools on two diffusion language models and shows that they can reveal meaningful features, such as patterns related to topics, languages, or styles. We also show that these features can be used to guide the model during its repeated refinement process, often more effectively than similar controls in standard language models. Beyond steering the output, the features help compare different orders for revealing words and mostly remain useful after the model is instruction-tuned. This work provides a first step toward making diffusion language models easier to inspect, control, and improve as they become more widely used.