Anatomy of Massive Activations and Attention Sinks

We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationships remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence and show that ablating it causes the two phenomena to decouple.

Large language models exhibit two puzzling behaviors: sometimes a few internal signals spike to extreme values ("massive activations"), and sometimes the model obsessively focuses on specific, seemingly unimportant words ("attention sinks"). Previously, it was observed that these phenomena frequently happen together, leaving their exact relationship unclear. To figure out what was really happening, we ran systematic experiments on the internal architecture of these models. We discovered that these two behaviors are actually distinct and serve different purposes. The massive spikes act like hidden global settings for the model , while the intense focus on specific words helps the model understand local sentence structure. The main reason they happen together is simply a side effect of a specific design choice used to scale down the model's numbers, known as normalization. By proving these behaviors can be separated, our research provides a clearer blueprint of how AI internal representations are shaped. This understanding will help engineers build more efficient, stable, and less memory-hungry AI models in the future, particularly for tasks requiring long text processing.