Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization

Multimodal hallucination remains a persistent challenge for Vision-Language Models (VLMs). Standard textual Direct Preference Optimization (DPO) often fails to mitigate it due to a lack of explicit visual supervision. While existing works introduce visual preference DPO by contrasting original images against negative ones, they suffer from a theoretically inconsistent objective caused by partition function mismatches and rely on coarse-grained negatives that could enable shortcut learning. In this work, we propose In-Context Visual Contrastive Optimization (IC-VCO). By placing contrastive images within a shared multi-image context, IC-VCO ensures a mathematically rigorous objective. We further introduce Visual Contrast Distillation (VCDist), an auxiliary reliability-gated regularizer that encourages consistency between multi-image contrastive training and single-image inference. Finally, we propose a contrastive sample editing strategy that generates hard negatives via precise semantic perturbations. Experiments on five benchmarks demonstrate IC-VCO's best overall performance and the effectiveness of our sample editing strategy. Code and data are available at https://github.com/OPPO-Mente-Lab/IC-VCO.

Vision-language models can answer questions about images, but they sometimes describe things that are not actually there or overlook small visual details. This matters because such systems may be used in visual assistants, education, and content-understanding tools, where plausible but wrong answers can mislead users. We study how to train these models to pay closer attention to the image, especially to fine-grained evidence such as an object’s color, existence, or spatial relation. Our method shows the model two highly similar images at the same time: the original image and a carefully edited version where only one important detail has changed. By asking the model to answer based on a specified image, training becomes a direct comparison of the visual detail that matters rather than a shortcut based on style or background differences. We also teach the model to carry this comparison skill back to the normal setting where it sees only one image. To create useful training examples, we use localized image edits instead of generating entirely new images, so the model must focus on subtle visual discrepancies. Across several benchmarks, this approach improves the reliability of vision-language models and helps reduce visually inconsistent answers.