Beyond Blind Noising: Disentangled Visual Rectification for Hallucination Mitigation in MLLMs

Visual Contrastive Decoding (VCD) mitigates hallucinations in Multimodal Large Language Models (MLLMs) by penalizing the output shift from noise-perturbed images, assuming this shift captures the hallucination direction. We prove this assumption flawed: noise-induced drift in Language-Image Pretrained (LIP) encoders is a \emph{coupled vector} entangling (i) structural degradation from corrupted visual information with (ii) hallucination induction from linguistic prior activation. VCD's indiscriminate penalty inevitably suppresses valid visual semantics. Our key insight is that Self-Supervised Learning (SSL) encoders exhibit \emph{only} structural degradation under noise—geometrically orthogonal to hallucination paths—enabling principled disentanglement via LIP--SSL differential response. We propose \textbf{Disentangled Visual Rectification (DVR)}, a training-free dual-stream framework performing visual-layer rectification and decoding-layer contrast on purified representations. DVR achieves approximately $5\times$ theoretical error reduction over VCD and establishes SOTA performance on POPE, MME, LLaVA-Bench and CHAIR benchmarks.

AI systems that process both images and text can "hallucinate"—generating descriptions of objects or events that do not actually appear in the image. A widely used remedy, Visual Contrastive Decoding (VCD), attempts to suppress these errors by adding noise to the input image and penalizing whatever changes in the model's output, under the assumption that such changes reflect hallucination tendencies. We demonstrate that this assumption is fundamentally flawed: the noise does not only trigger hallucinations but also destroys genuinely useful visual information. Penalizing both effects indiscriminately removes real content alongside fabricated content. Our key discovery is that a different class of vision models—trained through self-supervised learning rather than language-paired training—responds to image noise in a cleanly separable way, affecting only visual structure without triggering hallucinations. By comparing how these two types of vision models react to the same noise, we can precisely isolate and remove the hallucination component while preserving authentic visual understanding. Our resulting method, Disentangled Visual Rectification, requires no additional training and achieves roughly five times greater error reduction than VCD across major benchmarks.