Bigger Is Not Better: Inverse Scaling and Arbitration Failure in Counterfactual Visual Grounding

Images that violate everyday semantic expectations, such as a dog with five legs, expose a fundamental limitation of multimodal large language models (MLLMs): they may rely more on learned priors than on the visual evidence itself. Although prior work has observed such failures, we still lack a unified account of when they become more severe, why they arise, and how they can be mitigated. This work addresses all three questions. Across six MLLM families and three counterfactual visual grounding benchmarks, we find an inverse scaling trend: as models grow larger, they increasingly rely on language priors over visual evidence (when). Linear probing reveals that visual information is faithfully encoded throughout the LLM backbone, yet the model fails to act on it, reflecting an arbitration failure rather than a perceptual one. Further analysis shows that language priors are already committed in the early layers before visual signals begin to integrate in the middle of the network, suggesting that the arbitration imbalance must be corrected at the very first layers the LLM processes (why). We propose Visual Arbitration for MLLMs (VisArb), where a sparse autoencoder (SAE) is inserted between the vision backbone and the MLP connector, replacing dense feature maps with sparse, concept-aligned representations that are immediately legible to the arbitration mechanism from the first LLM layer onward (how). Our VisArb substantially improves counterfactual conflict recognition and reduces language-prior dominance while preserving performance on standard VQA benchmarks. Code is available here.