Inverse Depth Scaling From Most Layers Being Similar

Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics. This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.

Large language models (LLMs) like ChatGPT are built by stacking many identical-looking processing layers, but it is unclear whether adding more layers actually helps the model think more deeply or just adds redundancy. Understanding how depth contributes to performance matters for designing more efficient AI systems. We measured how the internal representations of text evolve as information flows through the layers of real LLMs, and found that the vast majority of layers make small, similar, and seemingly repetitive adjustments rather than building qualitatively new understanding. Fitting data from hundreds of models, we discovered a clean quantitative law: model error decreases inversely proportional to the number of layers, meaning doubling the depth only halves this particular source of error. We explain this law using a statistical argument: when many layers all attempt the same task with independent noise, their errors partly cancel, just as averaging many noisy measurements gives a more accurate result than any single one. This finding suggests that current LLMs use their depth inefficiently, operating more like a committee of similar experts voting together than like a chain of specialists each contributing unique expertise. To make future models more efficient, architectural innovations that encourage layers to build on each other in a genuinely hierarchical way, rather than redundantly averaging, could unlock substantially better performance per parameter.