MACD: Model-Aware Contrastive Decoding via Counterfactual Data for Video-LLMs

Video language models (Video-LLMs) are prone to hallucinations, generating plausible but ungrounded content when visual evidence is weak, ambiguous, or biased. Existing methods, such as contrastive decoding (CD), rely on random perturbations to construct contrastive data for hallucination mitigation, but often fail to target the visual cues that drive hallucination or align with model weaknesses. We propose Model-Aware Counterfactual Data based Contrastive Decoding (MACD), an inference strategy that combines model-guided counterfactual construction with contrastive decoding. MACD uses the Video-LLM's own feedback to identify object regions most responsible for hallucination, generating targeted object-level counterfactual inputs rather than arbitrary frame or temporal modifications. These counterfactual inputs are integrated into CD to enforce evidence-grounded token selection during decoding. Experiments on EventHallusion, MVBench, Perception-test, and Video-MME show that MACD consistently reduces hallucination while maintaining or improving task accuracy across diverse Video-LLMs, including Qwen and InternVL, with especially strong gains in scenarios involving small, occluded, or co-occurring objects.

Video-language models can answer questions about videos, but they sometimes describe things that are not actually present in the video. This problem is especially common when important visual evidence is small, unclear, or hidden by other objects. Our method, MACD, helps reduce these hallucinations by identifying which objects and frames are most important for the model’s answer, creating targeted counterfactual video inputs, and using them during decoding to favor answers that are better grounded in the original video. Across several video understanding benchmarks and model families, MACD improves factual reliability while keeping the model unchanged and requiring only a lightweight inference-time intervention.