Dissecting the Safety Circuit: Neuronal Intervention for Transferable Adversarial Attacks on VLMs

The limited transferability of adversarial attacks on Vision-Language Models (VLMs) stems from their failure to navigate model-specific safety alignments, where superficial perturbations exploit surrogate-specific artifacts rather than shared safety-critical features. We reveal through linear probing that safety-related representations are concentrated within specific intermediate neuronal circuits, which act as localized defense bottlenecks that can be disentangled from transferable features. To overcome this barrier, we propose the Safety Circuit Intervention Attack (SCIA), a framework that surgically steers internal representations to bypass these localized safety mechanisms. SCIA employs a dual-objective steering strategy that suppresses the defensive circuit encoding safety features while amplifying the transferable circuit capturing model-agnostic representations, effectively decoupling adversarial patterns from surrogate-specific safety behaviors. Furthermore, we incorporate contrastive semantic steering and spectral smoothness regularization to guide optimization toward compliant semantic regions while producing visually coherent perturbations. Experimental results demonstrate that SCIA significantly outperforms state-of-the-art methods in bypassing unseen black-box VLMs.

Vision-Language models are designed with safety mechanisms to block harmful outputs. However, attackers can craft subtle visual inputs to bypass these protections. Current bypass methods are often tied to the specific model they were developed on and transfer poorly to others. We discovered that safety protections inside these models are concentrated in small clusters of neurons rather than spread across the entire network. By identifying these clusters, we developed a method that manipulates them to bypass safety mechanisms not just in the original model, but also in unseen models, including commercial systems like GPT-4o and Gemini. These results show that current safety mechanisms can be circumvented by targeting a small number of neurons, exposing a concrete vulnerability that developers can address by redesigning how safety features are distributed within their models.