Securing Multimodal AI through Internal Information Decomposition

Multimodal large language models introduce attack surfaces absent in unimodal systems: adversaries can distribute malicious intent across modalities to evade unimodal safeguards. This motivates using cross-modal consistency as a detection signal rather than inspecting each modality in isolation. Our key observation is that benign inputs induce compatible predictive behavior from text-only and vision-only reasoning that stabilizes when fused, whereas adversarial manipulation disrupts this consistency, causing abnormal multimodal behavior. Existing defenses that examine raw inputs or outputs overlook this internal fusion process, rendering them brittle and computationally expensive. We propose FlowGuard, a lightweight inference-time framework that detects harmful inputs by monitoring internal multimodal consistency. Unlike approaches that rely on scalar confidence metrics, FlowGuard derives FlowVectors inspired by Partial Information Decomposition that quantify cross-modal redundancy, synergy, and modality-specific dominance, capturing whether multimodal fusion aligns with unimodal semantic evidencebetween unimodal and fused multimodal output distributions. In a one-class classification problem trained solely on benign data, FlowGuard reduces Attack Success Rates from $>90\%$ to $<15\%$ on unseen attacks, with $<3\%$ utility loss and up to a $6\times$ latency reduction. Our results demonstrate that monitoring cross-modal consistency offers an efficient and effective defense for multimodal reasoning.

Modern AI systems can answer questions about both images and text, but this also creates new safety risks. An attacker can hide part of a harmful instruction in an image and another part in the text, so that each part looks harmless on its own but becomes dangerous when the AI combines them. We developed FlowGuard, a method for detecting these attacks by looking at how a multimodal AI system combines visual and textual information internally. Instead of only checking the input image, the written prompt, or the final answer, FlowGuard compares what the model predicts from the text alone, the image alone, and both together. In normal cases, these three views tend to align together. When under attack, the combined prediction often behaves strangely, as if the image and text are pulling the model in conflicting or suspicious directions. FlowGuard learns what normal image-text interaction looks like using only safe examples, then flags unusual cases at inference time. Across several multimodal safety benchmarks, it substantially reduces successful attacks while preserving performance on benign tasks and adding relatively little computational cost. This suggests that monitoring how modalities are combined can make multimodal AI systems safer and more reliable.