Left–Right Symmetry Breaking in CLIP-Style Vision-Language Models Trained on Synthetic Spatial-Relation Data

Spatial understanding remains a key challenge in vision-language models. Yet it is still unclear whether such understanding is truly acquired, and if so, through what mechanisms. We present a controllable 1D image–text testbed to probe how left–right relational understanding emerges in Transformer-based vision and text encoders trained with a CLIP-style contrastive objective. We train lightweight Transformer-based vision and text encoders end-to-end on paired descriptions of one- and two-object scenes and evaluate generalization to unseen object pairs while systematically varying label and layout diversity. We find that contrastive training learns left–right relations and that label diversity, more than layout diversity, is the primary driver of generalization in this setting. To gain the mechanistic understanding, we perform an attention decomposition and show that interactions between positional and token embeddings induce a horizontal attention gradient that breaks left–right symmetry in the encoders; ablating this contribution substantially reduces left–right discrimination. Our results provide a mechanistic insight of when and how CLIP-style models acquire relational competence.

AI systems that combine vision and language do not always reliably understand spatial relationships like "the cat is to the left of the dog." Despite impressive object recognition abilities, these systems sometimes confuse left and right. Why this happens has remained unclear. We investigated how these systems learn to tell left from right by building a simplified setting where we could examine their internal workings. We discovered that the system develops an asymmetric attention pattern across an image, essentially learning to "look" more toward one side. Importantly, this pattern is not built in but emerges spontaneously as the system learns to generalize to new combinations of objects not seen during training. This pattern arises from how the system processes positional information. When we removed the positional components, the system could still recognize which objects were present but could no longer determine their spatial arrangement. Our findings reveal a specific internal mechanism responsible for spatial understanding in these AI systems. This insight could help researchers design more reliable systems that correctly understand spatial relationships, important for applications such as robotics, navigation, and visual assistance tools.