Towards Understanding Massive Activations in Attention Sink Mechanism

Recent studies have revealed two intriguing phenomena in large language models: attention sinks and massive activations. However, the co-emergence and co-existence of these two phenomena remain poorly understood. In this work, we revisit the prevailing view that massive activations are the primary mechanism responsible for concentrating attention on sink tokens, and provide a more nuanced interpretation of their relationship. Through both theoretical analysis and empirical evidence, we demonstrate that massive activations and attention sinks jointly act to prevent excessive token mixing in self-attention. Specifically, attention sinks suppress mixing among non-sink tokens, whereas massive activations suppress mixing between sink tokens and non-sink tokens. Furthermore, our theory provides a principled explanation of how KV-biases, gating mechanisms, and normalization layers can remove massive activations while largely preserving attention sinks. We further conduct intervention analyses and find that removing the value vector of the sink token can recover attention sinks even when massive activations are entirely suppressed. Overall, this work provides a mechanistic perspective on how massive activations and attention sinks interact under normalization and self-attention layers, offering new insights into their functional roles in Transformer models.

Large language models exhibit two interesting phenomena known as attention sinks and massive activations, but the relationship between them is still not well understood. Previous studies often suggested that massive activations are the main reason why models focus heavily on certain “sink” tokens. In this work, we show that the interaction is more subtle. Through theoretical analysis and experiments, we find that attention sinks and massive activations work together to control how information is mixed across tokens in self-attention. Attention sinks mainly reduce interactions among ordinary tokens, while massive activations reduce interactions between sink tokens and the rest of the sequence. We also explain why several architectural components in modern Transformer models, such as KV-biases, gating mechanisms, and normalization layers, can suppress massive activations without significantly weakening attention sinks. In addition, our intervention experiments reveal that removing the value vector of the sink token can restore attention sinks even when massive activations disappear completely. Overall, our work provides a clearer understanding of how these two phenomena cooperate inside Transformer models and offers new insights into the internal mechanisms of large language models.