The Abstraction Gap in Vision-Language Causal Reasoning

Vision-language models (VLMs) generate fluent causal explanations, but current evaluations cannot distinguish linguistic plausibility from faithful causal reasoning. We introduce a dual-probe methodology that isolates these properties. The Text-Only Probe measures linguistic quality. The Chain-Text Probe requires models to first generate explicit causal chains. The Abstraction Gap (AG) metric quantifies the normalized performance difference. Evaluating eight VLMs on CAGE (Causal Abstraction Gap Evaluation), a benchmark of 49,500 questions across 5,500 images spanning Pearl's causal hierarchy, we find seven models exhibit AG exceeding 0.50 with text scores of 6--8 but chain scores below 2.5. Fine-tuning on 45,000 chain-annotated examples fails to close the gap. However, one model achieves near-zero AG. The capability exists within current VLM architectures and depends on pretraining and architectural choices. CAGE provides a diagnostic tool for assessing faithful causal reasoning in VLMs.

AI systems that see and read images now work in high-stakes settings. They help doctors interpret medical scans. They help self-driving cars understand scenes. These vision-language models answer questions about cause and effect with confidence. But do they actually understand causation, or do they just sound like they do? We designed a simple test. Before explaining in sentences, show the reasoning steps by organizing them in the order that events occur, such as wind pushes umbrellas, umbrellas fall. Most systems could not do this. They wrote fluent explanations but failed to show even minimal step-by-step reasoning. Out of eight vision-language models tested, seven failed. We call this gap between fluent explanation and genuine reasoning the Abstraction Gap. To measure the Abstraction Gap systematically, we constructed CAGE (Causal Abstraction Gap Evaluation), a test with nearly 50,000 questions across thousands of images. CAGE identifies whether AI systems can show causal reasoning instead of just describing it. For medical diagnosis and autonomous vehicles, confident explanations without genuine understanding pose concrete risks. In our CAGE evaluation, most models failed, but one succeeded, which suggests the capability is achievable. The question is how to build more systems that truly understand.