Capacity without Access: Reinterpreting the Mid-Depth Spectral Plateau in LLMs

Prior probing-based analyses show that individual layers specialize in distinct linguistic and semantic functions, while a complementary line of work observes that deeper layers enter an alignment-dominated regime where residual transformations become increasingly constrained relative to the evolving hidden-state geometry. These findings appear to conflict: layer-wise functional specialization versus apparent representational stagnation in deeper layers. This raises a key question: Is the observed representational stagnation in deeper layers attributable to suboptimal acquisition or encoding of novel features, or are these features learned appropriately but exhibit minimal marginal contribution to the model’s predictive output? We address this by separating model behavior into (i) representational capacity: the richness and spectral diversity of the encoded features, and (ii) accessibility: the extent to which these features are aligned with, and exploited by, the output-relevant subspace. Analyzing hidden-state second-moment geometry across depth shows that intermediate layers maintain a broad representational span and rich spectral diversity, yet project only weakly onto the output-relevant subspace, indicating intact representational capacity but constrained accessibility within task-relevant dimensions. To probe their functional relevance, we add a controlled diagnostic pathway that routes intermediate-layer embeddings to the terminal readout. Analysis of the induced behavioral changes indicates that the apparent stagnation is better accounted for by geometric limitations on accessibility, rather than by a deficit in representational capacity. The source code is available here.

Large language models process text through many internal layers, but it is not always clear whether useful information inside these layers actually affects the final answer. Some layers may contain rich information that is only weakly used by the model when making predictions. This makes it hard to understand whether a model fails because it did not learn the right information, or because it learned the information but could not effectively use it. In this work, we study this distinction by separating what a model stores internally from how well that information is connected to its final prediction. We analyze the internal representations of several language models and introduce a diagnostic training method that temporarily gives intermediate layers a more direct path to the final output. After training, this extra path is removed, so the model keeps its original inference structure. Our results suggest that some mid-depth layers preserve useful information, but their influence on the final prediction can be limited. This provides a more precise way to understand how information is formed and used inside large language models.