Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Large Vision-Language Models (LVLMs) can answer questions about images and text, but they may still suffer from hallucination: producing responses that are inconsistent with the visual content. Our work studies why this happens and finds that LVLMs often pay insufficient attention to the correct visual evidence, and can gradually forget this evidence during generation. We also observe that different layers behave differently: some layers are more sensitive to the correct visual evidence, leading to clear Inter-Layer Visual Attention Discrepancy. Based on this finding, we propose Inter-Layer Visual Attention Discrepancy (ILVAD), a training-free and plug-and-play method for reducing hallucination. ILVAD collects attention weights from early generated tokens to visual tokens across layers, identifies visual tokens that are repeatedly activated, and uses them to build a saliency map. This saliency map helps enhance attention to visual evidence during generation, reducing visual forgetting. It also helps identify generated text tokens that are strongly grounded in visual evidence, so these tokens can be emphasized. Experiments on multiple benchmarks and five recently released LVLMs show that ILVAD consistently mitigates hallucinations across different model architectures.