When Does Sparsity Mitigate the Curse of Depth in LLMs

Recent work has demonstrated the curse of depth in large language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we provide evidence that sparsity-like mechanisms can dampen variance propagation and are associated with improved depth utilization Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and attention sparsity induced by long-context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing in Grouped-Query Attention and expert-activation sparsity in Mixture-of-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: mechanisms with reduced effective interaction density tend to exhibit lower output variance and better layer differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training depth-effective LLMs, yielding a notable 4.6 accuracy improvement on downstream tasks. Our results suggest that sparsity-like design choices are an important and previously underemphasized factor in effective depth scaling for LLMs. Code is available at https://github. com/pUmpKin-Co/SparsityAndCoD.

Large language models (LLM) are often made stronger by adding more layers, but recent studies show that very deep models may not use all of their layers effectively. In other words, some later layers behave almost like they are simply passing information forward, instead of learning useful representation transformations. This paper studies why this happens and finds that one important reason is the growth of activation variance as information moves through layers. We show that sparsity can help control this problem. Here, sparsity means that the model does not use all parameters, attention connections, or computational paths equally. We study both sparsity that naturally appears during training, such as from weight decay or long input sequences, and sparsity built into the model design, such as grouped-query attention and mixture-of-experts. Across many controlled experiments, these sparsity mechanisms with proper range reduce variance growth and make deeper layers more useful. Our findings suggest that sparsity is not only a way to save computation, but also a way to train deeper language models more effectively. By combining several sparsity mechanisms, we train a deeper model with better layer utilization and improvement in downstream accuracy.