Low-dimensional topology of deep neural networks

We study layered models, including feedforward networks, ResNets, and transformers, by limiting each layer to a width of $d = 3$, i.e., $\mathbb{R}^3$ as representation space. This allows us to track how a neural network changes low-dimensional topological invariants through its layers. Just about any topological structure may be simplified or even trivialized by simply increasing dimension; e.g., any knot is equivalent to an unknot in $\mathbb{R}^4$. By restricting to $\mathbb{R}^3$, we not only isolate the effects of activation and depth from that of width, we work in a space that lends itself to easy visualization. We focus on linking number here, deferring other invariants like link groups, Milnor's $\bar{\mu}$-invariants, knot types, ambient cobordisms, to a sequel. We provide full proofs and empirical experiments to justify the following insights: When measured by their power to effect changes in linking numbers, the layer-skipping feature in ResNets is as powerful as the attention mechanism in transformers; both ResNets and transformers are strictly more powerful than feedforward neural networks with monotonic activations, which are in turn more powerful than invertible and flow-based models; but replacing monotonic activation with a nonmonotonic one elevates a feedforward network into the same expressivity class as ResNets and transformers. These results suggest that low-dimensional topology can be a useful tool to guide designs of AI architectures. We also generalize our results from $d = 3$ to arbitrary $d > 3$.

Life ties itself in knots: proteins fold into tangled shapes, and vaccine mRNA coils into knot-like loops. Astonishingly, today's artificial intelligence faces the very same problem. The networks behind chatbots and image recognizers must tell one kind of data from another. Yet engineers found their best tricks long before they understood why. Picture two classes of data as interlocking rings. We prove a plain network can never pull them apart, no matter how deep it goes or how long it trains. The fixes are modern AI's own hallmarks: attention inside ChatGPT, the shortcut "residual connections" in deep networks, and newer "folding" activations. Together they flip space like a loop of string, until the rings come free. Our experiments confirm it, and we even find these tangles hiding in real photos. The lesson is a design principle: winning architectures fold and untangle the shapes hidden in data.