DREAM-MoE: Downstream Routing Error-Aware Margin-Preserving Quantization for Mixture-of-Experts Large Language Models

Post-training quantization (PTQ) is essential for deploying large Mixture-of-Experts (MoE) language models, but quantization can perturb router scores and change the selected experts. Existing router-aware quantization methods mainly align the router outputs of the quantized model with those of the full-precision model at the same layer. However, even small errors in matching router logits can change the relative ordering of experts and alter the selected top-$k$ set. Moreover, in block-wise PTQ where each transformer decoder block is quantized sequentially, reconstruction errors from the currently quantized decoder block can perturb the input to the following MoE router, leading to downstream routing changes. We propose \emph{DREAM-MoE}, a downstream routing error-aware margin-preserving objective for MoE quantization. DREAM-MoE preserves expert orderings that directly affect top-$k$ routing decisions by enforcing pairwise margins among selected experts and between selected experts and near-boundary unselected experts. It further regularizes each quantized decoder block using the downstream routing behavior of the next MoE router. DREAM-MoE is an auxiliary calibration objective and can be integrated into learning-based PTQ frameworks without adding inference-time modules or computational overhead. Across three MoE LLMs under 4-bit and 3-bit weight-only quantization, DREAM-MoE achieves the best average downstream accuracy in five of six model-bit settings and the lowest language-modeling perplexity in five of six settings.