Unveiling the Visual Counting Bottleneck in Vision-Language Models

While Large Vision-Language Models (VLMs) excel at interpolation, they suffer catastrophic failures in systematic generalization, most notably in visual counting beyond training distributions. In this work, we investigate this extrapolation bottleneck by deconstructing visual counting into three cognitive stages: object individuation, abstract magnitude representation, and symbolic decoding. Using a controlled environment of synthetic Go game boards, we isolate the specific mechanism of failure. Contrary to the hypothesis that models suffer from perceptual errors, we demonstrate via linear probing that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime. Furthermore, models retain latent magnitude awareness, successfully performing comparative reasoning on quantities they fail to enumerate. We pinpoint the collapse to the Symbolic Decoding stage, where the model fails to project valid visual magnitudes onto discrete tokens. Our findings support a Fractured Magnitude Hypothesis: VLMs fail to acquire a Universal Number Space, instead learning disjoint, modality-specific statistical manifolds that prevent cross-modal grounding for unseen pairings. We validate our findings on the state-of-the-art foundation model, suggesting that bridging the extrapolation gap requires inductive priors that enforce unified magnitude representations rather than simply scaling training data.

Artificial intelligence models that process both images and text are incredibly advanced, yet they share a surprising weakness: they fail catastrophically at counting objects if the amount is higher than what they were explicitly taught during training. Our research asks a simple question: why? Is the AI "blind" to the extra objects, does it lose track of the math, or does it just fail to attach the right word to the amount? By looking under the hood of these models using images of board games, we found that the AI is not blind at all: it perfectly perceives every individual object. It even understands the abstract concept of the quantity, successfully comparing amounts it hasn't seen before. The breakdown happens at the very last step. The AI completely fails to connect its accurate visual understanding of a number to the correct written word for it. The model essentially keeps its "visual counting" and "text counting" in two disconnected mental boxes. Our findings suggest that just feeding AI more data won’t solve this fundamental disconnect; instead, we must design models that inherently link seeing and speaking from the ground up.