Jailbreaking Vision-Language Models Through the Visual Modality

The visual modality of vision-language models (VLMs) is an underexplored attack surface for bypassing safety alignment. We introduce four jailbreak attacks exploiting the vision component: (1) encoding harmful instructions as visual symbol sequences with a decoding legend, (2) replacing harmful objects with benign substitutes (e.g., bomb $\rightarrow$ banana) then prompting for harmful actions using the substitute term, (3) replacing harmful text in images (e.g., on book covers) with benign words while visual context preserves the original meaning, and (4) visual analogy puzzles whose solution requires inferring a prohibited concept. Evaluating across six frontier VLMs, our visual attacks bypass safety alignment and expose a cross-modality alignment gap: text-based safety training does not automatically generalize to harmful intent conveyed visually. For example, our visual cipher achieves 40.9\% attack success on Claude-Haiku-4.5 versus 10.7\% for an equivalent textual cipher. To further our insight into the attack mechanism, we present preliminary interpretability and mitigation results. These findings highlight that robust VLM alignment requires treating vision as a first-class target for safety post-training.

AI assistants like ChatGPT, Claude, and Gemini now read pictures, not just text. They are trained to refuse harmful requests, but that safety training relies almost entirely on text examples. We asked whether a harmful request hidden in a picture could slip past these defenses. We designed four picture-based jailbreak attacks: one encodes a request using a legend of symbols; one swaps a dangerous object for a harmless stand-in; one removes a dangerous word from an image while keeping the visual context suggestive; and one hides the forbidden concept inside a visual riddle. We tested these on six leading AI models. Our results expose a gap: safety training on text does not automatically transfer to pictures. In one striking case, a visual cipher attack succeeded 41% of the time against a model that refused the matching text version 89% of the time. These attacks bypass the AI's input-side safety, but a text classifier scanning the reply catches the vast majority. We urge AI labs to deploy such a lightweight output classifier on every model. This low-cost defense closes most of the gap even under the compute pressures of today's AI race.