Beyond Logits: Coherent Hallucination Mitigation via Attention Contrastive Decoding

Large Vision-Language Models (LVLMs) demonstrate impressive multimodal capabilities, yet suffer from hallucination—generating factually inaccurate content. Contrastive Decoding (CD) mitigates this by contrasting amateur and expert branches at the logit level. However, our investigation reveals that such logit-level interventions fundamentally compromise generation coherence, necessitating restrictive penalty constraints unrelated to hallucination suppression. We introduce Attention Contrastive Decoding (ACD), a training-free plug-in that complements logit-level CD by relocating part of the contrastive operations to the attention mechanism. Operating at an earlier stage of the forward pass, ACD performs smooth semantic-preserving interventions through an Adaptive Subtraction Strategy (ASS), which attenuates hallucination-associated attention patterns while amplifying critical visual information. Extensive experiments demonstrate that combining ACD with existing CD methods (e.g., VCD+ACD) produces substantially more coherent outputs with further reduced hallucinations, eliminating restrictive penalties while enabling trustworthy multimodal generation.

AI systems that can understand both text and images sometimes "hallucinate"—they confidently describe things that are not actually present in an image. For example, when asked about a photo of a park, such a system might invent objects or events that do not exist in the scene. A popular fix works by comparing the outputs of a strong model against a deliberately weakened one, canceling out the patterns most likely to produce errors. However, we found that this comparison, applied at the final output stage, often disrupts the fluency and coherence of the generated text, requiring awkward workarounds that limit its effectiveness. Our method, Attention Contrastive Decoding, moves part of this correction to an earlier, internal stage of the model—where it decides what parts of the image to focus on. This produces smoother, more natural text while further reducing hallucinations. It requires no additional training and can be combined with existing approaches as a simple plug-in.