Quantifying LLM Attention-Head Stability: Implications for Circuit Universality

In mechanistic interpretability, recent work scrutinizes transformer “circuits”—sparse, mono or multi layer sub computations, that may reflect human understandable functions. Yet, these network circuits are rarely acid-tested for their stability across different instances of the same deep learning architecture. Without this, it remains unclear whether reported circuits emerge universally across labs or turn out to be idiosyncratic to a particular estimation instance, potentially limiting confidence in safety-critical settings. Here, we systematically study stability across-refits in increasingly complex transformer language models of various sizes. We quantify, layer by layer, how similarly attention heads learn representations across independently initialized training runs. Our rigorous experiments show that (1) middle-layer heads are the least stable yet the most representationally distinct; (2) deeper models exhibit stronger mid-depth divergence; (3) unstable heads in deeper layers become more functionally important than their peers from the same layer; (4) applying weight decay optimization substantially improves attention-head stability across random model initializations; and (5) the residual stream is comparatively stable. Our findings establish the cross-instance robustness of circuits as an essential yet underappreciated prerequisite for scalable oversight, drawing contours around possible white-box monitorability of AI systems.

Modern language models are often studied by looking for “circuits”: small groups of components inside a neural network that appear to carry out recognizable tasks, such as tracking syntax or recalling facts. These circuits are important for mechanistic interpretability because they may help researchers understand, monitor, and eventually make AI systems safer. But there is a key unanswered question: if two labs train the same model architecture from scratch, will they find the same circuits, or are the circuits partly an accident of one particular training run? This work studies that question by comparing independently trained transformer language models of different sizes. We measure how similarly attention heads—the model components that decide which words or tokens to focus on—learn representations across separate training runs. We find that attention heads in the middle layers are the least stable across runs and the most distinctive in their representations. This instability becomes stronger in deeper models, and in some deeper layers, the unstable heads are relatively more important for the model’s performance. At the same time, we find that weight decay, a common training technique, makes attention heads more stable across different random initializations. We also found that the residual stream, another major information pathway in transformers, is comparably more stable than attention heads. These results suggest that interpretability methods should not only identify circuits based on a single instance, but also test whether those circuits reliably appear across independently trained versions of the same architecture. This kind of robustness is especially important if we want to use interpretability for scalable oversight or safety monitoring of advanced AI systems.