On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression

Visual token compression is widely used to accelerate large vision-language models (LVLMs) by pruning or merging visual tokens, yet its adversarial robustness remains unexplored. We show that existing encoder-based attacks cannot fully disclose the robustness vulnerabilities of compressed LVLMs, due to an optimization-inference mismatch: perturbations are optimized on the full-token representation, while inference is performed through a token-compression bottleneck. To address this gap, we propose the Compression-AliGnEd attack (CAGE), which aligns perturbation optimization with compression inference without assuming access to the deployed compression mechanism or its token budget. CAGE combines (i) expected feature disruption, which concentrates distortion on tokens likely to survive across plausible budgets, and (ii) rank distortion alignment, which actively aligns token distortions with rank scores to promote the retention of highly distorted evidence. Across diverse representative plug-and-play compression mechanisms and datasets, our results show that CAGEconsistently achieves lower robust accuracy than the baseline. This work highlights that robustness assessments ignoring compression can be overly optimistic, calling for compression-aware security evaluation and defenses for efficient LVLMs.

Large vision-language models are powerful but expensive because they process many visual tokens from each image. A common way to speed them up is to compress these tokens before the model answers a question. However, we find that this compression can make models look more robust than they really are, because existing attacks are optimized for the full set of visual tokens rather than the compressed tokens used at inference time. We propose CAGE, an attack that aligns adversarial perturbations with the tokens that are likely to survive compression. Instead of spreading perturbations over all visual tokens, CAGE focuses on the compressed visual evidence that the model actually uses. Experiments across different compression methods, datasets, and model settings show that CAGE reveals stronger vulnerabilities than existing attacks. This work highlights an overlooked security risk in efficient vision-language models. As token compression becomes widely used to reduce inference cost, robustness evaluation must also account for the compressed inference process.