Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision-Language Models

Large Vision-Language Models (LVLMs) typically process visual inputs as a prefix to the language decoder. As the model autoregressively generates text, this initial visual information inevitably undergoes ``dilution'', leading the model to over-rely on language priors and hallucinate objects. Existing interventions attempt to correct this by contrasting logits or iteratively refining outputs, but they incur prohibitive latency costs. We propose **Residual-Update Directed DEcoding Regulation (RUDDER)**, a framework that counters visual dilution by creating a persistent visual anchor. We extract a robust evidence direction (**CARD**) directly from the model's prefill residual updates, and inject it into the decoding process. This injection is modulated by an adaptive gate, the **Beta Gate**, which acts as a trust mechanism and ensures the visual reminder is applied only when necessary. Experiments on LLaVA-1.5 (7B/13B), Idefics2, InstructBLIP, and Qwen2.5-VL demonstrate that RUDDER consistently mitigates hallucination (with greedy decoding, RUDDER reduces CHAIR$_S$ by an average of **24.4\%** and CHAIR$_i$ by **23.6\%** relative) and scales effectively across architectures, all while maintaining **>96.0\%** throughput. The code is available at https://github.com/Akko000/RUDDER-Residual-Update-Directed-DEcoding-Regulation-

Vision-language models can describe images and answer questions about them, but they sometimes mention objects that are not actually present. This problem is called object hallucination, and it makes model outputs less reliable. We propose RUDDER, a lightweight method that helps these models stay grounded in the visual information while generating text. RUDDER does not require retraining the model or running additional expensive forward passes. Instead, it extracts a visual anchor from the model’s own internal activations and uses it to guide generation. Experiments show that RUDDER reduces hallucinated objects across several vision-language models while keeping computational overhead very low.