DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning

Recent advances in multimodal language models (MLLMs) have made \emph{thinking with images} a dominant paradigm for multimodal reasoning. However, existing methods still fail to ensure \emph{evidence--answer consistency}, where correct answers must be supported by correct visual evidence. To address this issue, we propose \textit{DeFacto}, a counterfactual reasoning framework that explicitly aligns visual evidence with final answers. Our approach integrates three complementary training paradigms: (i) positive, (ii) counterfactual, and (iii) random-masking. We further develop a language-guided evidence construction pipeline that automatically localizes question-relevant regions and generates counterfactual variants, resulting in \textbf{DeFacto-100K}. Building on this dataset, we train MLLMs with GRPO-based reinforcement learning and design three complementary rewards to promote correct answering, structured reasoning, and consistent evidence selection. Moreover, we introduce \textbf{DeFacto-1.5K}, a human-annotated benchmark for systematically evaluating evidence-grounded consistency beyond answer accuracy. Experiments on diverse benchmarks demonstrate that \textit{DeFacto} substantially improves both answer accuracy and evidence--answer consistency over strong baselines. The code and datasets are available at \url{https://github.com/tinnel123666888/defacto}.

Modern AI systems that answer questions about images often produce confident answers that are not actually based on what is in the picture—they rely on clues from the question's wording or common-sense guessing rather than truly looking at the relevant visual region. In this work, we propose DeFacto, a training approach that teaches these systems to genuinely rely on visual evidence. Inspired by how people verify their reasoning by asking "what if I hadn't seen this part?", we train the model under three scenarios: when the key region is visible, when it is hidden so the model must admit it does not know, and when irrelevant parts are hidden so the model learns what truly matters. To support this, we automatically build a dataset of about 100,000 training examples and a smaller benchmark of 1,500 human-verified examples that tests whether a model's answers are properly grounded in the image. Across diverse evaluations, DeFacto produces more accurate and more trustworthy answers, is less likely to guess when key evidence is missing, and more often points to the correct visual region when answering correctly—moving the field toward AI systems that can not only give the right answer, but also show why it is the right answer.