Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility

Vision language models (VLMs) extend the reasoning capabilities of large language models (LLMs) to cross-modal settings, yet remain highly vulnerable to multimodal jailbreak attacks. Existing defenses predominantly rely on safety fine-tuning or \textit{aggressive} token manipulations, incurring substantial training costs or significantly degrading utility. Recent research shows that LLMs inherently recognize unsafe content in text, and the incorporation of visual inputs in VLMs frequently dilutes risk-related signals. Motivated by this, we propose Risk Awareness Injection (RAI), a \textit{lightweight} and training-free framework for safety calibration that restores LLM-like risk recognition by amplifying unsafe signals in VLMs. Specifically, RAI constructs an Unsafe Prototype Subspace from language embeddings and performs targeted modulation on selected high-risk visual tokens, explicitly activating safety-critical signals within the cross-modal feature space. This modulation restores the model’s LLM-like ability to detect unsafe content from visual inputs, while preserving the semantic integrity of original tokens for cross-modal reasoning. Extensive experiments across multiple jailbreak and utility benchmarks demonstrate that RAI substantially reduces attack success rate without compromising task performance.

Vision-Language Models (VLMs) can understand both images and text, but they currently have a significant blind spot: malicious users can easily bypass their text-based safety filters by hiding harmful requests (such as instructions for illegal activities) inside images or videos. Our research reveals that this vulnerability occurs because complex visual inputs inadvertently "dilute" the risk signals, confusing the AI's built-in safety mechanisms. To solve this problem, we developed a lightweight method called "Risk Awareness Injection" (RAI). Think of it as a pair of safety glasses for the AI. When the model receives an image or video, RAI immediately scans for hidden dangerous visual elements and explicitly amplifies these risk signals at the very first layer. This quickly awakens the AI's natural ability to recognize and refuse harmful content before it generates a response. Extensive testing shows that our approach successfully blocks these complex visual attacks without making the AI "dumb" on regular tasks—it still solves math problems and understands benign images perfectly. Importantly, RAI is highly efficient and does not require expensive retraining, making it a highly practical and accessible solution for deploying safer and more trustworthy AI systems in the real world.