Discovering Interpretable Failure Modes of Vision Language Models

Vision Language Models (VLMs) are increasingly deployed in safety-critical applications due to their general-purpose reasoning and adaptability with minimal domain-specific engineering. However, they pose catastrophic risks as they fail frequently in specific naturally-occurring scenarios, constituting failure modes. We present Revelio, a novel framework for the systematic discovery of interpretable failure modes in VLMs. We formally define a failure mode as a combination of interpretable, domain-specific concepts such as proximity of a pedestrian or weather conditions, for which a target VLM fails consistently. Discovering them requires efficiently searching an exponentially large, discrete combinatorial space. Revelio meets this challenge through two search strategies: a diversity-aware beam search for rapid mapping of the failure landscape, and a Gaussian-Process-based Thompson Sampling to globally explore complex failure modes. Applying Revelio to autonomous driving and indoor robotics reveals previously unknown vulnerabilities in state-of-the-art VLMs. In driving scenarios, VLMs lack spatial grounding and ignore major obstructions, suggesting actions that cause simulated collisions. In indoor settings, models either overlook hazards or exhibit overly cautious behaviors that trigger false alarms and degrade efficiency. By surfacing structured failure modes, Revelio provides developers with actionable diagnoses to guide targeted VLM safety remediations.