Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail

Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute, and performance. While these laws guide the development of modern foundation models, the mechanisms underpinning them remain poorly understood, in part due to the absence of scalable analysis tools. To close this gap, we introduce spectral position: a scalable measure of which eigenvalues of the empirical neural tangent kernel (eNTK) currently drive loss reduction. Applying this measure to scaling experiments, we find that spectral position decreases throughout training: learning shifts from dominant eigenmodes into the spectral tail. Larger models reach further into the tail than smaller models, revealing a size-dependent capacity we call spectral reach. This suggests why larger models achieve lower losses: they sustain learning on weak spectral signals inaccessible to smaller models. We further identify feature learning as a key enabler of spectral reach. It adaptively amplifies gradient magnitudes as learning advances, sustaining progress where frozen representations stall. This points to concrete interventions through architecture and optimizer design.

Modern AI models follow a striking pattern: the bigger the model and the more data you train it on, the better it performs. This relationship is so reliable that it now drives major decisions in AI development, such as how large an AI model should be and how much money should be spent on training it. Yet why this relationship happens remains poorly understood. To address this, we introduce a diagnostic that lets us watch how AI models learn during training. It scores how easy it is for the model to detect the pattern it is currently learning, on a scale from 1 (very obvious) to 0 (very subtle). We find that models start by learning simple, dominant patterns, then progressively shift toward subtler, finer-grained ones. Applying this diagnostic to a range of AI models, we find a consistent picture: bigger models reach further into these fine-grained patterns than smaller models can. We call this capacity spectral reach. It offers an answer to why larger models perform better: they aren't just doing the same thing slightly better; they can sustain learning on subtle patterns that smaller models cannot access at all. What enables this? As AI models train, they progressively build richer internal representations, a process called feature learning. We show that this process is an advantage: models that can't adapt their representations don't reach nearly as far. Our results suggest two concrete avenues for improvement: changing how AI models are designed and changing how they are trained. Both could help them reach deeper into the patterns that drive performance.