Learning Coherent Representations: A Topological Approach to Interpretability

Deep neural networks learn representations where individual features often lack interpretable meaning; a single neuron may activate for scattered, unrelated inputs. We introduce coherence, a geometric property inspired by neural coding in the brain, where neurons like grid cells and head direction cells respond to contiguous regions of state space. A non-negative matrix is coherent if each row (sample) attends to geometrically clustered columns (features) and vice versa, and in addition every sample is well described by some feature and every feature is needed by some sample. We prove that coherent matrices induce a bounded interleaving between the Vietoris-Rips filtrations of samples and features, guaranteeing that both spaces share compatible topological structure. This geometric constraint facilitates interpretability. For example, if data lies on a circle, coherent features must tile that circle into contiguous arcs. We introduce Coh, a differentiable objective function based on Fréchet variance that enforces coherence during training. Unlike sparsity, which bounds how many samples a feature activates on, coherence bounds which samples, requiring geometric connectivity rather than only rarity. This yields not just interpretable features but an interpretable feature space. We validate Coh in an auto-encoder using synthetic and rotated MNIST datasets and in a token embedding of BERT using language data.

AI systems learn by breaking data down into a set of internal "features." Ideally each feature would stand for one clear thing, but often a single feature reacts to a random mix of unrelated inputs, which makes it hard to tell what the system has actually learned. We borrowed an idea from the brain. Some neurons each respond to just one tidy part of the world — for example, one specific place a rat is standing in, or one direction it is facing. Because each neuron covers a single connected region, scientists can read off where the animal is from its brain activity. We wanted to make artificial networks organize themselves in the same neat way. To do that we define a property we call coherence. A representation is coherent when each feature responds to inputs that actually belong together, instead of scattered, unrelated ones. We show that when this holds, the data and the features describing it end up sharing the same overall shape. We also turn the idea into a simple addition that can be plugged into training. We tested it on simple shapes, rotated images of handwritten digits, and the word representations inside a small language model. In each case the method produced features that match recognizable concepts — like a feature that responds to one rotation of a digit, or to years, numbers, or place names. This is different from just making features fire rarely: rarity controls how often a feature is used, while coherence controls what it responds to. The payoff is features that are easier for people to make sense of, which is a small step toward AI systems we can actually inspect and understand.