CofactGVR: Counterfactual Intervention for Grounded Visual Reasoning

Despite rapid progress in Grounded Visual Reasoning (GVR) with MLLMs and RL-style fine-tuning, existing approaches often lack effective learning signals for intermediate grounding decisions and are prone to shortcut solutions. In this work, we explicitly decompose GVR into Evidence Generation followed by Counterfactual Answer Reasoning, and formalize this structure as a Causal Grounding Graph (CGG) in which the generated evidence acts as a causal mediator. Building on this formulation, we propose CofactGVR, which estimates the mediator’s utility via a matched counterfactual intervention that perturbs the predicted region while keeping the original image–question context fixed. The factual–counterfactual reward gap yields a principled intermediate bonus, selectively assigned to high-quality factual rollouts to promote evidence-faithful reasoning. To further stabilize and efficiently exploit this causal training signal, we incorporate a Quantile-filtered Prioritized Advantage Sampling (Q-PAS) strategy that preferentially updates on trajectories with high-magnitude advantages while filtering low-signal samples. Extensive experiments across GVR benchmarks show consistent improvements, indicating that CofactGVR strengthens reliance on informative visual evidence under controlled interventions. Our source code will be released at https://github.com/zhangy0822/CofactGVR.

AI systems are increasingly able to answer questions about images, but it is often unclear whether they truly use the relevant visual information or simply make a plausible guess based on the question and original images. For example, when asked a question about an object in a picture, a model may give the right answer even if it has not actually looked at the right object. This makes such systems harder to trust, especially in tasks where users need to know why an answer was produced. This paper introduces CofactGVR, a training method that encourages an AI model to base its answers on the image regions that really matter. The key idea is simple: after the model chooses a visual region as evidence, we ask what would happen if it were forced to look somewhere else. If its answer becomes worse, then the chosen region was likely useful evidence. The model is then rewarded for relying on such meaningful evidence. Experiments on several benchmarks show that CofactGVR improves both answer accuracy and the model’s ability to use relevant visual evidence. This makes visual reasoning systems more reliable.