Formalizing the Binding Problem

Representations of the world, arguably, contain information about features (e.g. something is blue, something is a circle) but also information about which features are part of the same object (e.g. the circle is blue), which we call binding information. Any system with the ability to understand scenes with multiple objects must be able to solve the binding problem: it needs to know which features belong together. However, despite work showing that Vision Transformers (ViTs) know which patches belong together, it is not known whether current deep learning models learn to exhibit binding information, i.e., for features. We may believe that there is not much binding information, after all misattributing features to wrong objects is a common failure of ViT-based architectures, especially in scenes with objects sharing features. Here we formalize the binding problem with an information-theoretic approach, and introduce a probing method to measure binding information in model representations. We perform experiments on ViTs, measuring binding from different components of the architecture, such as the image summary token [CLS] or the spatial tokens. We use datasets with different binding challenges, such as feature sharing, occlusion, and natural features, while comparing the performance of several pre-trained ViTs. Overall, our research demonstrates binding as a key ingredient to strong visual recognition and reasoning.

People can usually tell which visual details belong together: for example, that the red object is a square and the blue object is a circle, rather than mixing up the colors and shapes. Many AI vision systems still struggle with this, especially when scenes contain multiple similar, overlapping, or cluttered objects. This paper studies how well AI image models keep track of which features belong to which objects. We introduce a way to measure this ability inside a model, rather than only judging whether the model’s final answer is correct. We find that a model’s single overall summary of an image often loses much of this object-level information. In contrast, the model’s separate spatial image pieces preserve much more information about which objects are present. Our results show that object-feature binding is an important part of strong visual understanding, and that improving it may help future AI systems reason more reliably about complex scenes.