Seeing Without Understanding: Disentangling Perception, Reasoning, and Simulation in VLM Gameplay

While Vision-Language Models (VLMs) excel on static visual benchmarks, they consistently underperform in game-based reasoning, yet existing evaluations conflate failures in perception, rule comprehension, and reasoning. We propose a two-stage diagnostic framework that decomposes VLM performance into testable components: controlled perception tests isolating visual encoding, and a diagnostic matrix with a six-level rule complexity ladder evaluated in both explicit verification and predictive simulation modes. Experimenting with six state-of-the-art VLMs reveals three failure patterns: (1) coordinated spatial drift, where off-by-one localization errors among adjacent pieces share the same shift direction at $1.5$-$1.9\times$ the rate expected under spatial independence; (2) perception-reasoning dissociation, where models correctly verify board states but fail to apply rules—at complex constraint levels, perception remains relatively stable while reasoning accuracy plummets, with even the best-performing model capped at $75\%$; and (3) a simulation gap, with performance dropping by up to $27$ points when predicting future states versus verifying observed outcomes. These limitations persist across model scales and are not resolved by scaling, text-only input, or structured prompting. Code and data are available at https://github.com/chillibeaver/PRS-Diag.

AI systems that process both images and text can describe photos and answer visual questions, but they struggle with tasks that require understanding rules and reasoning about what they see. When these systems play board games like chess, they make mistakes, but until now it was unclear whether they fail because they cannot see the board correctly, cannot understand the rules, or cannot think ahead. We designed controlled tests to separate these abilities. Surprisingly, when these systems misread a game board, the errors are not random: neighboring pieces tend to be shifted together in the same direction, as if entire regions of the image slide as a block. We also found that even when the systems read the board perfectly, they still fail to apply game rules correctly, and that they can verify whether a completed move was legal much more easily than they can predict whether a future move is possible. Using more powerful versions of these systems or giving them step-by-step instructions does not fix these problems. These findings matter beyond games. Any application where AI needs to look at structured visual information and reason about it, from reading documents to guiding robots, may face the same blind spots.