The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity

Despite the prevalence of the attention sink phenomenon in Large Language Models (LLMs), where initial tokens disproportionately monopolize attention scores, its structural origins remain elusive. This work provides a mechanistic explanation for this phenomenon. First, we trace its root to the value aggregation process inherent in self-attention, which induces a systematic variance discrepancy. We further demonstrate that this discrepancy is drastically amplified by the activation of super neurons within Feed-Forward Network (FFN) layers. Specifically, the channel-sparse down-projections trigger a dimension disparity of the first-token representation, necessitating the formation of attention sinks as a structural anchor. Then, we validate this causal chain through two controlled interventions: (i) isolating the aggregation effect via attention mask modifications and (ii) amplifying the variance of targeted token representations. Both interventions can replicate attention sinks at arbitrary positions. Our mechanistic understanding offers a foundation for the systematic control of sink formation. Finally, as a proof of concept, we propose head-wise RMSNorm, an architectural modification that stabilizes value aggregation outputs during pre-training. Our experiments demonstrate that restoring statistical parity across positions significantly accelerates convergence. The code is available at https://github.com/Siquan-Li/Head-wise-RMSNorm.

Large language models process text by deciding which earlier pieces of text are most useful for understanding the next one. Surprisingly, they often give unusually high importance to the very first piece of text, even when it has no special meaning. This paper explains why that happens. We show that the first piece is treated differently from later pieces: it only uses its own information, while later pieces mix information from previous context. This small imbalance is amplified as the model processes the text, eventually turning the first piece into an artificial anchor. We test this explanation by moving the special treatment to other positions, and the same anchor behavior moves with it. Based on this understanding, we add a simple balancing step inside the model so that different positions and information channels stay on a comparable scale. In training experiments, this reduces the unwanted anchor effect, preserves richer internal representations, and helps the model learn faster. Overall, the work gives a clearer explanation of a puzzling behavior in language models and suggests a practical way to make them more stable.