Unifying Low Dimensional Spectra in Deep Learning

Low-dimensional structures appear ubiquitously in the eigenspectra of deep-learning matrices in classification networks trained in the overparameterized regime. While theoretical advances have aimed to explain this phenomenology, they typically succeed only in capturing subsets of the full behavior or rely on assumptions that cannot hold in practice. In this work, we provide an analytic explanation for the bulk–outlier structure of several canonical deep-learning matrices, including the Hessian, gradients, and weights. We achieve this using unconstrained feature models (UFMs), a now-common tool for studying the emergence of deep neural collapse (DNC). We show that DNC is the source of these low-dimensional eigenspectra: in each case, the eigenvalues and eigenvectors can be constructed from feature means, the characterizing objects of DNC. This provides a unifying analytic explanation for a wide range of spectral phenomena in deep learning and goes beyond empirical characterizations—which typically focus on eigenvalues—by providing a detailed analysis of eigenvectors. We prove that our results hold for both linear and ReLU networks and provide numerical validation in both the modeling context and standard deep-network architectures on canonical datasets.

Modern neural networks often contain millions of adjustable numbers, making it hard to understand why training works or what the network has learned. One recurring clue is that several important mathematical objects inside trained networks, such as curvature, gradients, and weights, show a surprisingly simple structure: most directions seem unimportant, while a small number carry most of the signal. This paper explains where that simplicity comes from. We show that it is caused by a phenomenon called deep neural collapse, where examples from the same class become grouped together inside the network, and different classes become arranged in a highly organized geometric pattern. Using a simplified but widely studied model of overparameterized neural networks, we prove that this class geometry determines the important directions in the Hessian, gradients, and weights. We further confirm the predictions in experiments on standard image-classification models. These results give a unified explanation for several previously separate observations about neural-network training. They suggest that studying and controlling deep neural collapse could provide a more interpretable way to understand training, diagnose models, and design networks with better behavior.