On the Intrinsic Limits of Transformer Image Embeddings in Non-Solvable Spatial Reasoning

Vision Transformers (ViTs) excel in semantic recognition but exhibit systematic failures in spatial reasoning tasks such as mental rotation. While often attributed to data scale, this work argues that the limitation arises from the intrinsic circuit complexity of the architecture. By formalizing spatial understanding as a **Group Homomorphism Problem**—where latent embeddings preserve the algebraic structure of physical transformations acting on images—we identify a fundamental computational bottleneck. Specifically, for non-solvable groups (e.g., $\mathrm{SO}(3)$), maintaining such structure-preserving embeddings is lower-bounded by the Word Problem, which is $\mathsf{NC^1}$-complete. In contrast, constant-depth ViTs with polynomial precision are strictly bounded by the complexity class $\mathsf{TC^0}$. Under the standard conjecture $\mathsf{TC^0} \subsetneq \mathsf{NC^1}$, a **complexity boundary** emerges: constant-depth architectures lack the logical depth required to capture non-solvable spatial structures in a single forward pass. To empirically validate this theoretical gap, we propose the **Latent Space Algebra (LSA)** benchmark, which reveals a significant degradation in ViT representations as the compositional depth of non-solvable tasks increases.

AI vision systems have transformed how computers recognize objects and scenes, but they consistently struggle with a different class of tasks: reasoning about how objects look after being rotated or repositioned in three-dimensional space. This failure is widely assumed to stem from insufficient training data or model size. This work demonstrates that the limitation is instead fundamental to how these architectures compute. The core argument connects two areas of mathematics. Physical transformations such as 3D rotations have an algebraic structure where the order of operations matters in a deep, non-decomposable way—rotating an object left then up produces a different result than up then left, and this entanglement cannot be simplified or parallelized. Resolving a long sequence of such transformations requires computation that scales with the length of the sequence. Vision Transformers, however, have a fixed computational depth that does not grow with the number of transformations being reasoned about, making them structurally ill-suited for the kind of sequential, order-dependent computation that 3D spatial reasoning demands. Under a widely accepted conjecture in theoretical computer science, this architectural constraint makes it provably impossible for such models to faithfully represent 3D rotational structure, regardless of how many parameters they contain. To validate this theoretically derived boundary, we construct a benchmark that tests whether model representations preserve the mathematical structure of transformations at three levels of complexity: 2D translations, 2D rigid motions, and 3D rotations. Across all tested architectures and model sizes, performance degrades sharply and consistently on the 3D rotation tasks, while remaining stable on simpler ones. Increasing model depth provides negligible improvement, confirming that the barrier is structural rather than a capacity issue. These findings carry practical implications for applications requiring reliable spatial understanding, such as robotics and autonomous navigation, where failure under novel spatial sequences could have safety consequences. They also suggest that continued investment in scaling existing architectures for spatial tasks may be misdirected, and that progress requires fundamentally different computational designs.