Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

Vision language models (VLMs) often generate hallucination, i.e., content that cannot be substantiated by either textual or visual inputs. Prior work primarily attributes this to over-reliance on linguistic prior knowledge rather than visual inputs. Some methods attempt to mitigate hallucination by amplifying visual token attention proportionally to their attention scores. However, these methods overlook the visual attention sink problem, where attention is frequently misallocated to task-irrelevant visual regions, and neglect cross-modal fusion balance by enhancing only visual attention without adjusting attention to the user query. This can result in amplifying incorrect areas while failing to properly interpret the user query. To address these challenges, we propose a simple yet effective method called Gaze Shift-Guided Cross-modal Fusion Enhancement (GIFT). GIFT pre-computes a holistic visual saliency map by tracking positive changes in visual attention, or "gaze shifts", during user query comprehension, and leverages this map to amplify attention to both salient visual information and the user query at each decoding step. This reduces the impact of visual attention sink, as irrelevant tokens exhibit minimal shifts, while ensuring balanced cross-modal fusion for well-integrated representation. Extensive experiments show that GIFT effectively mitigates hallucination in VLMs across both generative and classification tasks, achieving up to 20.7% improvement over greedy decoding, while maintaining general vision-language performance with low computational overhead.

Vision-language models can describe what they see in an image and answer questions about it, but they often "hallucinate", i.e., confidently stating things that aren't actually in the picture. For example, if asked about a tennis match shown in a photo, a model might invent a player who isn't there. This happens because the model's attention drifts onto irrelevant parts of the image and over-relies on what it has memorized from text rather than what the image actually shows. We took inspiration from how human eyes work. When people read a question about a picture, their gaze shifts toward the parts of the image that matter for answering. We taught the model to track its own internal "gaze shifts" while reading the question, building a map of which image regions are truly relevant. The model then pays more attention to those regions and to the question itself when generating its answer. Our method reduces hallucinations by up to 20.7% across several models and tasks, without retraining and with minimal extra computation, helping make vision-language systems more trustworthy in settings like medicine, self-driving, and robotics.