DLLMQuant: A Post-Training Quantization Framework Tailored for Diffusion-Based Large Language Models

Diffusion-based large language models (DLLMs) have shown promise for non-autoregressive text generation, but their deployment is constrained by large model sizes and heavy computational costs. Post-training quantization (PTQ), a widely used method for compressing and accelerating Large Language Models (LLMs), suffers from severe accuracy degradation and reduced generalization performance when directly applied to DLLMs (e.g., AWQ suffers a 16% accuracy drop on LLADA under W4A4). This paper explores how the unique mechanisms of Dynamic Language Models (DLLMs) conflict with quantization, identifying three core issues: 1) During the iterative generation process of DLLMs, dynamic masking ratios are inherently involved, leading to notable differences in token distributions across decoding steps. Unfortunately, these distinct distributions are not sufficiently captured by current PTQ calibration approaches; 2) Quantization errors propogate and accumalte progressively during iterations in DLLMs, leading to a gradual decline in the performance of quantized models as decoding steps advance; 3) The stability of unmasked tokens, combined with the probabilistic nature of masked tokens, gives rise to an overall feature distribution that is uncoordinated and unsuitable for PTQ. To address these issues, we propose DLLMQuant, a PTQ framework tailored for DLLMs, which incorporates three novel techniques: 1) Temporal-Mask Adaptive Sampling (TMAS), a calibration method that accounts for both time and mask factors, with the capacity to capture distributions across timesteps. 2) Interaction-Aware Activation Quantization (IA-AQ), which utilizes bidirectional attention scores to identify important tokens, and prioritizes these tokens when minimizing quantization error. 3) Certainty-Guided Quantization (CGQ) incorporates mask status and token scores as core weighting criteria for error compensation, enabling PTQ to better align with the unique weight distribution of DLLMs. Experiments show that DLLMQuant achieves significant performance gains (e.g., over 10-point accuracy improvement on GSM8K for LLADA under 4-bit quantization) while enhancing efficiency.

New AI text generation models that work through iterative diffusion processes can produce text faster than traditional models, but they are very large and require huge computing resources to run, which greatly limits their practical real-world use. A common, simple method to shrink and speed up AI models usually fails badly on these new diffusion-based text models, causing major drops in their accuracy and adaptability. In this work, we figured out three key reasons why this standard model compression method does not work for diffusion text models. First, these models generate text step by step with changing token selection rules at every step, creating varying data patterns that standard compression methods cannot fully adapt to. Second, small errors produced during model compression keep building up through each step of text generation, making the model’s output quality worse and worse the longer it generates text. Third, the mixed stable and uncertain features of the text being generated create inconsistent data patterns that conflict with conventional compression techniques. To solve these problems, we designed a new dedicated compression tool specifically for diffusion-based text models. This tool uses three targeted improvements to fit the unique working rules of these models: it adapts to the changing rules of each generation step, prioritizes protecting key text information to avoid quality loss, and adjusts compression errors based on the certainty of generated content. Our tests prove that this new compression method greatly preserves and even improves the model’s performance after compression—with large accuracy gains on common reasoning tasks—while successfully making the model more efficient and faster to run.