GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs

Mixture-of-Experts Large Language Models (MoE-LLMs) achieve strong performance but incur substantial memory overhead due to massive expert parameters. Mixed-precision quantization mitigates this cost by allocating expert-wise bit-widths based on their importance, approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization. However, existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization, resulting in suboptimal allocation and routing. In this work, we propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations via (1) a global linear-programming formulation that captures model-wide expert importance based on quantization error analysis, and (2) efficient router fine-tuning to adapt routing to quantized experts. These components are integrated into a progressive quantization framework that iteratively refines importance estimation and allocation. Experiments demonstrate that GEMQ significantly reduces memory and accelerates inference with minimal accuracy degradation.

Machine learning systems such as large language models are becoming increasingly powerful, but running them requires a large amount of memory and computing resources. This is especially true for Mixture-of-Experts (MoE) models, which contain many specialized components (“experts”) that make the models efficient in theory but expensive to deploy in practice. Existing compression methods reduce memory usage by storing different experts at different numerical precisions, but they often make decisions locally within each layer and fail to account for how quantization changes the model’s routing behavior. In this work, we introduce GEMQ, a new compression framework for MoE language models. GEMQ first estimates the importance of experts across the entire model rather than layer by layer, and then uses optimization techniques to allocate precision more effectively. It also adapts the model’s routing mechanism after quantization so the model can better work with compressed experts. Finally, GEMQ progressively refines these decisions through multiple stages. Our method significantly reduces memory usage and speeds up inference while maintaining strong accuracy. This can help make large AI models cheaper and more accessible to deploy on limited hardware.