Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding

MLLMs frequently hallucinate objects inconsistent with visual inputs. This issue is typically attributed to the over-reliance on language priors, which can override the visual context. Recent training-free decoding strategies address this by penalizing language priors. However, these methods overlook the dual nature of language priors, where they can be both helpful and harmful depending on the alignment with visual evidence. In particular, blindly suppressing language priors often disrupts the model’s semantic manifold, leading to performance degradation, a phenomenon we term Manifold Departure. To address this, we propose Manifold-Guided Adaptive Projection (MGAP), a geometry-aware, training-free decoding method that mitigates hallucinations while preserving representation structure. MGAP first constructs a language-prior subspace from blind hidden states via SVD. During decoding, MGAP projects each multimodal hidden state onto this subspace and applies a consistency-aware gate to adaptively attenuate only the projected prior component, yielding a subspace-selective update that largely preserves the orthogonal semantic components. Extensive experiments on POPE and CHAIR show that MGAP outperforms prior decoding baselines, achieving stronger hallucination suppression without sacrificing coherence.

In recent years, AI models that combine vision and language—known as Multimodal Large Language Models—have become powerful tools for understanding the world. However, these models frequently struggle with 'hallucinations,' where they confidently describe objects or scenes that do not actually exist in the images provided. This issue limits their reliability in safety-critical areas like medical diagnostics or autonomous driving. Our research identifies that this problem often stems from the models blindly relying on their internal, biased expectations about how the world should look, rather than focusing on the actual image. To solve this, we developed a new decoding method called MGAP. Instead of simply blocking or suppressing these internal expectations, which can make the model less accurate, our method allows the AI to dynamically adjust its focus. It selectively ignores biased information only when it conflicts with what is clearly visible in the image. This ensures the AI remains both helpful and accurate, producing descriptions that are truly grounded in evidence. Our approach makes these AI systems safer, more trustworthy, and better suited for real-world applications where accuracy is paramount.