ImpQuant: Fine-Grained Importance-Aware Quantization for Large Vision-Language Models

Large Vision–Language Models (LVLMs) have demonstrated remarkable capabilities across diverse multimodal tasks, yet their high inference costs necessitate low-bit deployment. Existing post-training quantization (PTQ) pipelines primarily adopt methodologies from text-only LLMs by treating multimodal inputs as homogeneous sequences, overlooking the heterogeneous information density inherent in LVLMs. In this work, we present ImpQuant, an importance-aware PTQ framework tailored for LVLMs that mitigates low-bit accuracy degradation via fine-grained token-importance reweighted calibration and outlier-aware activation quantization. Our key insight is that quantization errors on decision-critical tokens disproportionately impact overall model behavior. Accordingly, we reweight the calibration loss using aggregated attention for textual tokens and a contextual redundancy metric for visual tokens, respectively. Across multiple LVLM backbones and diverse multimodal benchmarks, our approach consistently improves accuracy at low bitwidth and reduces quantization-induced object hallucinations compared to state-of-the-art PTQ baselines.

Modern AI systems that answer questions about images can be very powerful, but they are often too large and slow to run cheaply on everyday hardware. A common way to shrink them is to store their numbers with less detail, but this can make them miss small visual clues or even mention objects that are not in the image. We introduce ImpQuant, a compression method designed for image-and-language AI systems. Instead of treating every word and every image patch as equally important, ImpQuant asks which parts are most likely to affect the answer and protects those parts more carefully during compression. It also handles unusually large internal signals separately so compressing them do not distort the rest of the model’s information. Across several popular image-question answering models and benchmarks, ImpQuant keeps compressed models more accurate than existing methods and reduces false object claims caused by compression. This makes large image-and-text AI systems easier to deploy while preserving the visual details that matter for reliable answers.