Singular Vectors of Attention Heads Align with Features

Identifying feature representations in language models is a central task in mechanistic interpretability. Several recent studies have made the observation that feature representations can be inferred in some cases from singular vectors of attention matrices. However, sound justification for this phenomenon is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we demonstrate that singular vectors robustly align with features in a model where features can be directly observed. We then show theoretically that such alignment is expected under a range of conditions. We close by asking how, operationally, alignment may be recognized in real models where feature representations are not directly observable. We identify sparse attention decomposition as a testable prediction of alignment, and show evidence that it emerges in real models in a manner consistent with predictions. Together these results suggest that alignment of singular vectors with features can be a sound and theoretically justified basis for feature identification in language models.

As language models become more widely used, understanding how they represent concepts is increasingly important for controlling them and improving their safety. For example, when a model analyzes an input like “the city of Dallas,” how is that notion actually stored in the model? The field has put forward many methods for finding those representations, but none so far are entirely satisfactory. This paper takes a new look at the problem and shows that, in many cases, a small number of directions already present in the model’s parameters (its stored weights) line up with the concepts the model uses. This gives us a way to analyze the model’s parameters to identify how concepts are represented. This approach has several advantages over previous methods: it is potentially more efficient and works directly with the model’s own weights, making it more closely tied to how the model itself is organized. Working on this problem matters because if we can better identify the features models use, we can better understand what is happening under the hood and build tools that help us control, audit, and improve systems that increasingly shape how people work, learn, and communicate.