How can embedding models bind concepts?

Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision–language embedding models such as CLIP struggle with binding: they recognize individual concepts but fail to represent which concepts form which objects. Although CLIP behaves like a bag-of-concepts model in cross-modal retrieval, object information is recoverable from its image and text embeddings separately. We study this tension through the binding function, which maps concepts to scene embeddings. We find that scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover object information. However, CLIP’s binding function is high-complexity, which likely prevents the image and text encoders from learning a shared binding mechanism that generalizes to unseen concept combinations. We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer models trained from scratch, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions characterized by multiplicative interactions between concepts, enabling systematic generalization.

When we see a red square next to a blue circle, we instantly know which color belongs to which shape. AI systems that link images and text, such as CLIP, are surprisingly bad at this. They can recognize "red", "blue", "square", and "circle" in a picture, but often cannot tell us that the square is red, not blue. We asked why. We found that the rule these systems use to combine features into objects is needlessly complicated -- they essentially memorize each combination separately, rather than following a simple shared pattern. Because the rule is so complicated, the image side and the text side of the model end up using different, incompatible versions of it, and they disagree about combinations they have not seen before. We then showed this is not an unavoidable limit. When we trained smaller AI systems from scratch on carefully designed data, they discovered a simple, reusable rule that behaves like a multiplication of feature representations, and they correctly recognized brand-new combinations. The takeaway: the problem with models like CLIP is not what these systems know, but how they organize it. Training future systems to learn simpler, more reusable rules could fix one of their most stubborn weaknesses.