Grokking Finite-Dimensional Algebra

This paper investigates the grokking phenomenon, which refers to the sudden transition from a long memorization to generalization observed during neural networks training, in the context of learning multiplication in finite-dimensional algebras (FDA). While prior work on grokking has focused mainly on group operations, we extend the analysis to more general algebraic structures, including non-associative, non-commutative, and non-unital algebras. We show that learning group operations is a special case of learning FDA, and that learning multiplication in FDA amounts to learning a bilinear product specified by the algebra’s structure tensor. For algebras over the reals, we connect the learning problem to matrix factorization with an implicit low-rank bias, and for algebras over finite fields, we show that grokking emerges naturally as models must learn discrete representations of algebraic elements. This leads us to experimentally investigate the following core questions: (i) how do algebraic properties such as commutativity, associativity, and unitality influence both the emergence and timing of grokking, (ii) how structural properties of the structure tensor of the FDA, such as sparsity and rank, influence generalization, and (iii) to what extent generalization correlates with the model learning latent embeddings aligned with the algebra’s representation. Our work provides a unified framework for grokking across algebraic structures and new insights into how mathematical structure governs neural network generalization dynamics.

Sometimes, when learning, children do not seem to understand something at first; they simply mimic what they see. But after enough repetition, something clicks: they suddenly “get it” and can apply the idea in new situations. The same thing can happen with artificial intelligence (AI). During their training, AI models often start by memorizing the training examples. Yet, after a surprisingly long number of training steps/time, they begin (suddenly) to understand the underlying patterns and solve problems they have never seen before. This sudden shift is called **grokking**. Most previous studies of grokking focused on very simple mathematical problems, such as modular arithmetic or basic group operations. These are systems with very regular and predictable rules. For example, in modular arithmetic, numbers “wrap around” like on a clock: after 12 comes 1 again, so on a 12-hour clock, 10 + 5 becomes 3, not 15. Similarly, group operations describe systems in which combining objects follows strict, consistent rules, much like repeatedly applying moves on a Rubik’s Cube or rotating a shape. In these systems, the rules are stable: combining objects in different ways still produces consistent outcomes, there is always a neutral element that leaves objects unchanged (like adding zero or multiplying by 1), and operations can be grouped without changing the result (like $(2 + 3) + 4 = 2 + (3 + 4)$). Because these problems are highly structured and relatively simple, they provide a clean environment for studying how AI models move from memorization to genuine understanding. In this paper, we investigate whether the same phenomenon occurs in a much broader family of mathematical systems, namely finite-dimensional algebras. These systems allow a much wider variety of behaviors than the simpler mathematical rules usually studied in grokking. In some of these systems, combining objects in different orders can produce different results. In others, even the way operations are grouped can matter. For example, in ordinary arithmetic, $(2 \times 3) \times 4 = 2 \times (3 \times 4)$, but in more general systems this property may fail. Some systems may also lack a simple “neutral element,” such as zero for addition or one for multiplication. Similar mathematical structures naturally appear in quantum physics, chemistry, robotics, computer graphics, and natural language processing. One motivation for studying these systems comes from language itself. The meaning of a sentence often depends on the order in which words are combined: “dog + bites + man” and “man + bites + dog” contain the same words but mean very different things. Likewise, in chemistry, combining substances in different sequences can lead to different outcomes. These examples suggest that understanding richer mathematical interactions may help explain how AI systems learn complex structures in the real world. We introduce a unified mathematical framework that shows that many previously studied grokking tasks are special cases of this larger setting. Instead of treating each mathematical problem independently, we describe them using mathematical objects called structure tensors, which encode the rules governing how elements interact with one another. This allows us to compare many different systems within a single framework. Using neural network experiments, we study how different mathematical properties influence learning. We find that some algebraic structures are much easier for AI models to generalize from, while others significantly delay grokking. We also observe that successful generalization is strongly connected to the emergence of meaningful internal representations inside the neural network. Before grokking, the model mostly memorizes examples individually. After grokking, its internal representations become organized in a way that reflects the hidden mathematical structure of the task, allowing it to correctly solve new examples it has never encountered before. Our results suggest that the mathematical structure of a problem plays a major role in how AI systems learn and generalize. More broadly, this work helps connect abstract mathematics with modern AI research and provides new insight into why neural networks sometimes suddenly transition from memorization to understanding.