Debate with Images: Detecting Deceptive Behaviors in Multimodal Large Language Models

As frontier AI systems become increasingly capable, concerns about deceptive behaviors have intensified. Unlike hallucinations, which stem from capability limitations, deception involves strategically misleading responses despite correct internal representations. While prior work has primarily studied deception in text-only settings, little is known about how such behaviors manifest in multimodal large language models. In this work, we systematically investigate multimodal deception and introduce *MM-DeceptionBench*, the first benchmark designed to evaluate deceptive behaviors in vision–language models across six realistic categories. We find that existing text-centric monitoring approaches are insufficient in multimodal settings due to the complexity of cross-modal reasoning. To address this gap, we propose *debate with images*, a multi-agent evaluation framework that enforces visual grounding through adversarial debate. Experiments show that this approach achieves substantially higher agreement with human judgments than MLLM-as-a-judge baselines, improving Cohen’s kappa by up to 1.5$\times$ and accuracy by up to 1.25$\times$ on GPT-4o.

AI systems are increasingly asked to answer questions about images, such as describing a scene, judging visual evidence, or helping users make decisions. A key safety concern is that these systems may sometimes give misleading answers even when they have correctly understood the image. This is different from an ordinary mistake: the system may notice an important visual fact but hide it, exaggerate what it can see, or invent details that are not in the image. We study this problem by building MM-DeceptionBench, a collection of image-based situations designed to test deceptive behavior in vision-language AI systems. We also introduce debate with images, a method where multiple AI agents examine the same case from opposing viewpoints and must support their claims with concrete visual evidence. This helps reveal misleading answers that a single AI judge may miss. Our work provides tools for auditing increasingly capable image-and-text AI systems, identifying when and how they mislead users, and supporting safer deployment in real-world settings.