From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning

Visual causal reasoning is essential for understanding and intervening in the physical world, requiring identification of causal variables from visual inputs and reasoning over intervention effects. Despite recent progress, large vision-language models (VLMs) remain brittle at such tasks, especially for interventional and counterfactual queries over multi-image inputs. Most existing explorations inject causal knowledge via textual prompts, leaving causal mechanisms external to model execution and limiting reliable control during inference. To address this problem, we propose BridgeVLM, which internalizes visual causal reasoning by inducing a causal graph from multi-image inputs and converting it into structured Causal Tokens executed by RAMP layers injected into the LLM decoder for causal message passing. We further introduce a unified training interface M3S for fine-grained causal supervision from different granularities (local/global level). BridgeVLM achieves 54.4\% accuracy on intervention tasks on CausalVLBench (vs. 33.2\% with prompt-level supervision), improves results on Causal3D from 43.6\% to 49.0\%, and substantially improves causal structure learning on CausalVLBench ($F_1$: 33.4\% $\rightarrow$ 75.1\%).

AI systems that can look at images and answer questions have become increasingly capable, but they still struggle with cause-and-effect reasoning. For example, if two images show the same scene before and after a change, the system may need to identify what was changed and predict how that change should affect other parts of the scene. Current models often receive this kind of causal information only as long written prompts, but they may not reliably connect those instructions to the visual evidence. We introduce BridgeVLM, a method that gives such models an internal representation of the important variables in an image and how they are related. Instead of treating causal knowledge only as text, BridgeVLM uses it to shape internal representations that the model can consult when answering questions. This makes the model better at intervention and counterfactual reasoning across several visual causal reasoning benchmarks. Our results show that BridgeVLM improves over strong baselines, especially when causal supervision is provided internally rather than only through prompts.