Deep sequence models tend to memorize geometrically; it is unclear why

Deep sequence models are said to store atomic facts predominantly in the form of *associative* memory: a brute-force lookup of co-occurring entities. We identify a dramatically different form of storage of atomic facts that we term as *geometric* memory. Here, the model has synthesized embeddings encoding novel *global* relationships between all entities, including ones that do not co-occur in training. Such storage is powerful: for instance, we show how it transforms a hard reasoning task involving an $\ell$-fold composition into an easy-to-learn $1$-step navigation task. From this phenomenon, we extract fundamental aspects of neural embedding geometries that are hard to explain. We argue that the rise of such a geometry, as against a lookup of local associations, cannot be straightforwardly attributed to typical supervisory, architectural, or optimizational pressures. Counterintuitively, a geometry is learned even when it is more complex than the brute-force lookup. Then, by analyzing a connection to Node2Vec, we demonstrate how the geometry stems from a spectral bias that—in contrast to prevailing theories—indeed arises naturally despite the lack of various pressures. This analysis also points out to practitioners a visible headroom to make Transformer memory more strongly geometric. We hope the geometric view of parametric memory encourages revisiting the default intuitions that guide researchers in areas like knowledge acquisition, capacity, discovery, and unlearning.

An important aspect of AI models is their "memory" of facts i.e., its ability to answer highly specific questions like "which country is adjacent to what?", without you giving it clues in its context. What we want to ask is, how does the model store such details? One form of storage that researchers have proposed is something called "associative memory". In this view, the belief is that the model stores these facts as a lookup table i.e., a table where one column is a country and the second column is an adjacent country. We point out that there is a very different form of storage that we call "geometric memory". In a geometric memory, the model has developed its own "global" atlas of sorts, where the model assigns each country a carefully-calculated "location" that is close to adjacent countries. This sort of highly-organized atlas is not specified in the raw data (of "local" adjacencies) the model was trained on, but emerges naturally. We show how this can enable the model to reason and plan over its facts, make novel connections not in the training set (e.g., there is a path from Country A to Country D!). Such powerful things are not as feasible under a lookup-like storage. We also show what this atlas is: it is tied to some mathematical properties inherent to the underlying associations specified in the training data. This is the "eigenvectors" of the adjacency matrix. We also point out a puzzle here (hence the "it's unclear why"). If you ask an ML scientist, why the model uses a geometric storage rather than the associative/lookup storage, they'd probably point to a few culprits e.g., the lookup is perhaps too cumbersome/complex for the model to capture (an atlas seems simpler to remember, after all!); or that the lookup is perhaps too complex to "find" as the model searches over all possible solutions to fit the data. We design experiments refuting these answers. So, why does the model memorize facts geometrically rather than associatively? As of now, we don't know, but we hope a future paper will help understand this.