Beyond Fixed Biases: Decoding the Role of Reasoning Uncertainty in MLLM Modality Conflicts

Multimodal Large Language Models (MLLMs) must resolve conflicts when modalities provide contradictory information, a behavior we term "modality following". We propose a framework that decomposes this behavior into case-specific relative preference uncertainty and stable inherent preference. Across diverse MLLMs and benchmarks, the probability of following a modality consistently decreases as its relative preference uncertainty increases, a trend robust to alternative uncertainty indices. This regularity defines a "balance point'' where modality preferences are evenly matched, offering a capability-disentangled measure of modality bias. Layer-wise probing further shows that ambiguous cases near the balance point trigger middle-to-late-layer "concept oscillations," where top predictions vacillate between modality-supported answers. Finally, we demonstrate the framework's utility for preference steering through Supervised Fine-Tuning (SFT). We find that data efficiency is governed by preference uncertainty: training on easy samples (where one modality dominates) fails to generalize, whereas targeting the identified ``boundary cases" is essential for robust preference alignment and suppressing internal vacillation.

Multimodal AI systems use both images and text, but these sources can sometimes disagree. This paper studies how such systems decide which source to trust. We find that their choices depend not only on a fixed preference for image or text, but also on how uncertain the model is about each source in each specific case. When both sources are similarly convincing, the model often becomes internally unstable, switching between the two possible answers. We further show that training on these difficult boundary cases is more effective than training on easy examples for improving how models handle conflicting information.