Recursive Binding on a Budget: Subspace Carving in Order-$p$ Tensor Memories

Tensor Product Representations provide the structural fidelity required for symbolic reasoning in models but suffer from *exponential* dimensionality growth when encoding deep recursive structures. Conversely, Vector Symbolic Architectures maintain *constant* dimensionality but sacrifice capacity and fidelity due to noisy compression via superposition. In this work, we propose **Orthogonal Subspace Carving (OSC)**, a memory architecture that binds *fillers* to *roles* by projecting onto the null space of the role basis before aggregating into a fixed order-$p$ tensor. OSC uses projections to enforce geometric orthogonality between bound structures within a {\em static} memory trace. We show that this mechanism decouples the tensor order from the structural depth, enabling deep recursive binding within a *constant* memory footprint. This construction allows for component vectors that are *orders of magnitude* smaller than the memory tensor, giving excellent memory efficiency in settings involving high superposition. We also show that TPR is a special case of binding in Clifford algebra, and give a Clifford formulation of OSC.

Computers struggle to remember and reason about structured information, things like family trees, sentences, or nested instructions, in a way that scales. One existing approach stores structure faithfully but the memory needed grows explosively with depth, so it quickly becomes impractical. Another approach keeps memory small but introduces so much noise that retrieval becomes unreliable. We propose Orthogonal Subspace Carving (OSC), a new way to store structured information that escapes this tradeoff. Our key idea is to represent each piece of information as a collection of smaller vectors rather than a single large one. The collection together spans an enormous representational space, but the cost of storing any individual piece stays small. The price we pay is that we can no longer reconstruct stored items from scratch. Instead, we recognize them by comparing candidates against memory, much like a multiple-choice test rather than a fill-in-the-blank. However, the original architectures introduced so much noise that the multiple-choice answers were needed anyway as the write-ins were too fuzzy to read on their own. This means machine learning systems can represent and manipulate structured knowledge, like the steps of a long argument or the relationships in a complex document, far more efficiently than before, opening the door to symbolic reasoning capabilities in models that previously could not afford them.