Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time. However we found that during autoregressive decoding, visual conditioning affects token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical tokens, and can cause instability. To address these limitations, we propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation. Our approach first extract token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity–adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse vision-language models. It modulates the steering strength at each decoding step, selectively suppressing hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal and HallusionBench, demonstrating consistent improvements over previous steering methods.

Large vision-language models often generate hallucinated content that is not supported by the input image. This is partly because they may rely too much on language priors and fail to sufficiently use visual evidence. Activation steering is a lightweight inference-time control method that guides model behavior by injecting a vector into the model’s activation states. We use this idea to encourage multimodal models to rely more on image information and reduce hallucinations. Our analysis shows that visual dependence is highly sparse at the token level: most tokens are only weakly affected by the image, while a small number of critical tokens strongly depend on visual evidence. Existing methods usually perform sentence-level steering with a fixed injection strength, which can dilute important visual signals and disturb non-critical tokens. We therefore propose Token-Level Visual-Sensitivity Steering (TLVS). TLVS extracts steering directions from visually sensitive tokens and dynamically adjusts the injection strength during generation. This allows the model to apply stronger visual grounding only where it matters. We also optimize the method to reduce inference overhead, making TLVS an efficient and plug-and-play approach for mitigating hallucinations in large vision-language models.