Mechanistic Analysis and Inference-Time Control of Modality Conflict in VLMs

When a vision-language model sees an image that contradicts a textual statement, which modality does it trust, and can that preference be controlled? Although prior work has answered this question behaviorally, we answer it mechanistically and causally. Behaviorally, we introduce CLEVR-CONFLICT, a controlled dataset of multimodal conflicts with verified ground truth, and show that LLaVA-v1.6 overwhelmingly follows text (9.2% vision-following), while Qwen2.5-VL predominantly follows vision (80.7%). Mechanistically, contribution patching identifies the bridge module as the causal entry point for visual information, and logit lens locates a midnetwork commitment in LLaVA and a sharp latelayer flip in Qwen. Causally, MLP replacement at the commitment zone flips 73.3% of Qwen’s textfollowing entries and 23.4% of LLaVA’s, with the gap reflecting a distributed-vs-concentrated architectural distinction. We develop a stateless offset vector that achieves bidirectional inferencetime control, pushing toward vision or toward text without retraining. Causal tests further reveal the MLP is necessary but not sufficient. It enforces language priors only when conflicting text has activated them, functioning as a context-dependent amplifier rather than a standalone switch. Our findings generalize to natural images from COCO. Together, the findings offer a mechanistic account of modality conflict that moves from diagnosis to causal characterization to inference-time control.