VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models

Vision-Language Models (VLMs) extend large language models with visual reasoning, but their multimodal design also introduces new, underexplored vulnerabilities. Existing multimodal red-teaming methods largely rely on brittle templates, focus on single-attack settings, and expose only a narrow subset of vulnerabilities. To address these limitations, we introduce VERA-V, a variational inference framework that recasts multimodal jailbreak discovery as learning a joint posterior distribution over paired text-image prompts. This probabilistic view captures complex cross-modal interactions, enabling stealthy, coordinated adversarial inputs that bypass model guardrails. We train a lightweight attacker to approximate the posterior, allowing efficient sampling of diverse jailbreaks and providing distributional insights into vulnerabilities. VERA-V further integrates three complementary strategies: (i) typography-based text prompts that embed harmful cues, (ii) diffusion-based image synthesis that introduces adversarial signals, and (iii) structured distractors to fragment VLM attention. Experiments on HarmBench and HADES benchmarks show that VERA-V consistently outperforms state-of-the-art baselines on both open-source and frontier VLMs, achieving up to 53.75\% higher attack success rate (ASR) over the best baseline on GPT-4o. We include the code on the project page available here: https://github.com/kxwhiowo/VERA-V

Vision-Language Models (VLMs) that process both text and images are powerful but carry hidden security vulnerabilities. Current methods for discovering these weaknesses are often rigid, focusing on simple, isolated attacks that fail to capture the complex, malicious ways images and text can interact to bypass safety filters. We developed VERA-V, a probabilistic framework that learns how text and visual inputs interact to generate highly coordinated jailbreak attacks. By using a lightweight attacker guided by model feedback, VERA-V creates composite prompts which combine hidden harmful text, subtle adversarial images, and visual distractors to confuse VLMs. Our research proves that VLMs are highly susceptible to coordinated, cross-modal attacks. By systematically exposing these complex blind spots and outperforming existing baselines, VERA-V provides researchers with a concrete, automated tool to rigorously stress-test and strengthen the next generation of multimodal AI defenses.