Vision Language Models Cannot Reason About Physical Transformation

Understanding physical transformations is fundamental for reasoning in dynamic environments. While Vision Language Models (VLMs) show promise in embodied applications, whether they genuinely understand physical transformations remains unclear. We introduce ConservationBench evaluating conservation—whether physical quantities remain invariant under transformations. Spanning four properties with paired conserving/non-conserving scenarios, we generate 23,040 questions across 112 VLMs. Results reveal systematic failure: performance remains near chance with improvements on conservation tasks accompanied by drops on controls. Control experiments show strong textual priors favoring invariance, yet models perform worse with visual content. Neither temporal resolution, prompting, nor curated sampling helps. These findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.

A key sign of human intelligence is understanding that certain physical properties stay the same even when things look different—knowing that spreading coins apart doesn't change how many there are, or that pouring water into a taller glass doesn't give you more water. Children typically develop this understanding, called conservation, around age 7, and it forms a cornerstone of how we reason about the physical world. We asked: do today's most capable AI vision systems have the same ability? To find out, we built ConservationBench—a collection of real-world videos showing everyday physical transformations across four properties: number, length, volume, and size. Crucially, each video is paired with a matched "trick" version where the quantity actually does change, letting us check whether models are genuinely reasoning or just guessing. Testing 112 Vision Language Models (VLMs)—AI systems that process both images and text—we found consistent failure across the board. Rather than watching what happens and reasoning about it, models simply default to assuming quantities stay the same. This works half the time but fails badly on the trick videos. Giving models more video frames, better prompts, or larger model sizes didn't help. When we removed the visual content entirely, models still answered confidently based on language patterns alone—and surprisingly, adding real visual content actually made things worse. Internal analysis revealed that models fixate on the first frame and fail to track how objects change over time. These findings reveal a fundamental gap in how current AI systems understand the physical world—one that matters for any application requiring reliable reasoning about real-world events and transformations.