VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets, but we still do not fully understand how they perform under real-world image distortions. We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometric perturbations, evaluated under graded severities (low/mid/high) and binary transforms, yielding 133 corrupted settings. We evaluate 15 VLMs from five families, including proprietary GPT-5.4-mini, on two complementary benchmarks: MMBench (visually grounded) and MMMU-Pro (reasoning-oriented). Our results reveal that visual severity is a weak predictor of difficulty: low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions. In particular, low-severity glass blur reduces MMBench accuracy by about 8 pp on average across models, while the largest drops arise from resampling and geometric distortions (e.g., upsample, elastic transform), reaching up to 34 pp. Overall, our findings suggest current VLMs are semantically strong but spatially fragile, motivating the definition of novel robustness evaluation protocols and training regimes that emphasize resampling and geometric invariance.

AI systems that understand both images and text, known as vision-language models, are increasingly used in applications such as transport, healthcare, robotics, and document processing. These systems perform impressively on standard tests with clean, well-composed images. But real-world images are rarely perfect: cameras produce noise in low light, rain or fog can obscure the scene, photos may be compressed, and images can be resized, blurred, or rotated. This raises a simple question: do these systems still work reliably when visual inputs are imperfect? We introduce VLM-RobustBench, a large-scale test of 15 leading vision-language models under 49 types of image distortion, covering effects such as blur, noise, weather, compression, resizing, and geometric changes. Our main finding is that how distorted an image looks to a human does not reliably predict how much it confuses an AI model. Some mild changes, such as resizing, slight blur, or flipping an image upside down, can reduce accuracy more than visually dramatic distortions such as strong noise or compression. This weakness appears across model families, including very large models. Overall, today's vision-language models are often strong at understanding clean images, but can be surprisingly fragile when images deviate from familiar visual patterns. This matters for safety-critical settings such as driving, robotics, and medical imaging, where small image changes that humans would easily tolerate may lead to sharp drops in model performance.