Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs). However, this approach has significant limitations as it neglects the geometric properties of the value vectors being aggregated. To address this gap, we introduce \emph{Contribution Weights}, a projection-based metric that quantifies a token's influence by accounting for it's attention weight, value magnitude, and directional alignment with the layer output. We demonstrate that contribution weights provide a more faithful measure of token importance, consistently outperforming attention-based metrics in identifying semantically critical tokens across different decoder-only models, tasks, and datasets. Further, our metric enables novel mechanistic analysis of \emph{attention sinks}. While previous work characterized sinks as passive repositories for excess attention, we reveal they serve an active functional role, suppressing information through a convex relationship between sink rate and output norm, stabilizing representations by opposing the semantic drift of low-confidence tokens.

The problem. To interpret how language models work, researchers typically examine "attention weights"—a measure of how much each word focuses on others—and assume that high attention indicates high importance. We find this assumption misleading. The contribution. We introduce Contribution Weights, which incorporate two factors that attention alone omits: the magnitude of a word's signal, and the direction in which it pushes the output (reinforcing it, opposing it, or neither). This yields a more faithful account of which words genuinely matter, and the result holds across several models when the metric is validated by removing words it deems unimportant. Key insight. Models concentrate substantial attention on the first token, long regarded as a passive "parking spot" for excess attention. We demonstrate that it serves an active function: its signal points opposite to the meaningful content, acting as a stabilizing brake that cancels accumulated noise. Why it matters. Evaluating models by attention alone can lead to incorrect conclusions; accounting for signal magnitude and direction provides a more faithful interpretation.