Global Geometry Is Not Enough for Vision Representations

A common assumption in representation learning is that globally well-distributed embeddings support robust and generalizable representations. This focus has shaped both training objectives and evaluation protocols, implicitly treating global geometry as a proxy for representational competence. While global geometry effectively encodes which elements are present, it is often insensitive to how they are composed. We investigate this limitation by testing the ability of geometric metrics to predict compositional binding across 21 vision encoders. We find that standard geometry-based statistics exhibit near-zero correlation with compositional binding. In contrast, functional sensitivity, as measured by the input-output Jacobian, reliably tracks this capability. We further provide an analytic account showing that this disparity arises from objective design, as existing losses explicitly constrain embedding geometry but leave the local input-output mapping unconstrained. These results suggest that global embedding geometry captures only a partial view of representational competence and establish functional sensitivity as a critical complementary axis for modeling composite structure.

A vision model that has seen many photographs learns an internal "map" of what it has seen, where every image becomes a point in a high-dimensional space. For years, researchers have judged how good such a model is by checking properties of that map: whether the points spread out evenly, whether they avoid collapsing onto a single line, whether the geometry looks well-balanced. This kind of geometric tidiness has become a standard benchmark for representation quality. We tested whether geometric tidiness actually predicts something more demanding: a model's ability to tell apart scenes that have the same parts arranged differently. A nut on top of a washer and a washer on top of a nut share the same ingredients but mean different things. To pass our test, a model must encode not just what is present but how things are composed. Across various widely-used visual encoders, geometric tidiness did not predict this compositional ability. Models with beautifully isotropic, well-spread maps could still fail at composition. Instead, we found that a different quantity, how sensitively the model's output responds to changes in its input, closely tracks compositional ability. We give this quantity a precise definition (the "Jacobian Effective Rank") and show that it is shaped, in predictable ways, by the choice of training objective: some objectives constrain it directly, others leave it largely free to drift. The practical upshot is twofold. First, the geometric scorecard the community uses to evaluate representations captures only part of what matters; compositional capability needs a complementary diagnostic. Second, this diagnostic is concrete enough to be used today: to compare encoders before deploying them, to guide the design of new training objectives, and to identify which layers of a network are responsible for losing structural information.