Semantic Robustness Certification for Vision-Language Models

Vision-language models (VLMs) are now widely used in downstream tasks. However, real-world applications often expose VLMs to distribution shifts induced by semantic variation (e.g., shape, size, and style). Robustness certification determines if a model’s prediction changes when transformations are applied to its input. While most certification frameworks study geometric or pixel-level transformations over inputs, this work proposes a novel framework that enables certifying VLM robustness under semantic-level transformations. Leveraging the open-vocabulary capability of VLMs, we use text prompts as semantic proxies to construct transformations parameterized by an extent that controls the degree of semantic variation. By characterizing the VLM decision boundary in closed form, our framework quantitatively certifies extent intervals for which the predicted class remains unchanged under the semantic transformation. Our framework is the first to certify VLM robustness under semantic-level variations without requiring additional data for each variation, making it practical to apply. Experiments on both synthetic and real-world data show that our framework enables certifying robustness under diverse semantic variations across scenarios.

Vision-language models are increasingly used to connect images with language, but their predictions can be sensitive to visual changes such as shape, size, style, viewpoint, or background. This paper develops a method for determining when a model’s prediction remains stable under such semantic variations. Using text descriptions to specify the intended change, the method constructs a controlled semantic path and identifies the ranges where the prediction is guaranteed to remain unchanged. Experiments on generated and real images show that the method captures prediction changes across diverse semantic variations, providing a practical way to assess the robustness of vision-language models under semantic changes in visual content.