VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models

Mixture-of-Experts(MoE) Vision-Language Models(VLMs) offer remarkable performance but incur prohibitive memory and computational costs, making compression essential. Post-Training Quantization (PTQ) is an effective training-free technique to address the massive memory and computation overhead. Existing quantization paradigms fall short as they are oblivious to two critical forms of heterogeneity: the inherent discrepancy between vision and language tokens, and the non-uniform contribution of different experts. To bridge this gap, we introduce Visual Expert Quantization (VEQ), a dual-aware quantization framework designed to simultaneously accommodate cross-modal differences and heterogeneity between experts. Specifically, VEQ incorporates 1)Modality-expert-aware Quantization, which utilizes expert activation frequency to prioritize error minimization for pivotal experts, and 2)Modality-affinity-aware Quantization, which constructs an enhanced Hessian matrix by integrating token-expert affinity with modality information to guide the calibration process. Extensive experiments across diverse benchmarks verify that VEQ consistently outperforms state-of-the-art baselines. Specifically, under the W3A16 configuration, our method achieves significant average accuracy gains of 2.04\% on Kimi-VL and 3.09\% on Qwen3-VL compared to the previous SOTA quantization methods, demonstrating superior robustness across various multi-modal tasks.

Modern AI models that understand both images and text have become remarkably capable, but they are enormous and require expensive hardware to run. A popular design for making them more efficient splits the model into many specialized sub-networks and activates only a few at a time, yet even these models are still too large for most everyday devices. A common way to shrink them is to store their internal numbers with less precision, similar to rounding decimals to fewer digits. The catch is that existing shrinking methods treat the whole model uniformly, ignoring two important facts: images and text are processed very differently inside the model, and some experts contribute far more than others. Rounding everything the same way damages the model's accuracy. We built a method called VEQ (Visual Expert Quantization) that respects these differences. It identifies which experts matter most based on how often they are used and whether they specialize in images or text. So we can protect those experts during compression. It also takes into account how strongly each piece of input is tied to each expert, so the compression effort is focused where it matters. Across many image-and-text tasks, VEQ produces compressed models that are 2–3% more accurate than the best previous methods. This makes powerful multimodal AI cheaper to run on smaller devices, and our code is publicly available for others to build on.