HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson’s paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to encourage learning from internal evidence and combines it with a learned prior over external features to recover the true posterior. While intervention-based mitigation methods often degrade utility or fluency by modifying models' internals, we use HaloProbe as an external scoring signal for non-invasive mitigation. Our experiments show that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art intervention-based methods while preserving caption utility and linguistic fluency.

Vision-language models can describe images in natural language, but they sometimes mention objects that are not actually present. For example, a model might describe a “cat” in an image that contains no cat. These hallucinations reduce the reliability of AI systems and can be problematic in applications that require accurate visual understanding. Many existing approaches attempt to detect hallucinations by analyzing how much attention the model pays to image regions. We show that these signals can be misleading because they are influenced by hidden factors such as where an object appears in a sentence and whether it has been mentioned before. These factors can create statistical effects that lead to incorrect conclusions about model behavior. To address this issue, we introduce HaloProbe, a Bayesian framework that combines information from the model’s internal decision process with simple statistics about the generated description. HaloProbe estimates the probability that each mentioned object is correct and uses these estimates to reduce hallucinations without modifying the model itself. Experiments across several vision-language models show that HaloProbe detects hallucinations more accurately than previous methods and reduces them more effectively while preserving the quality and fluency of generated descriptions.