Mechanistic Interpretability as Statistical Estimation: A Variance Analysis

Mechanistic Interpretability (MI) aims to reverse-engineer model behaviors by identifying functional sub-networks. Yet, the scientific validity of these findings depends on their stability. In this work, we argue that circuit discovery is not a standalone task but a statistical estimation problem built upon causal mediation analysis (CMA). We uncover a fundamental instability at this base layer: exact, single-input CMA scores exhibit high intrinsic variance, implying that the causal effect of a component is a volatile random variable rather than a fixed property. We then demonstrate that circuit discovery pipelines inherit this variance and further amplify it. Fast approximation methods, such as Edge Attribution Patching and its successors, introduce additional estimation noise, while aggregating these noisy scores over datasets leads to fragile structural estimates. Consequently, small perturbations in input data or hyperparameters yield vastly different circuits. We systematically decompose these sources of variance and advocate for more rigorous MI practices, prioritizing statistical robustness and routine reporting of stability metrics.

As AI systems are increasingly used in high-stakes settings, researchers have been trying to understand what is actually happening inside them. One popular approach is to identify small groups of connections inside a neural network, called "circuits", that are responsible for specific behaviors (e.g. solving mathematical tasks). But how trustworthy are these circuits? We tested whether circuits found by standard methods stay the same under small, reasonable changes: using slightly different data, tweaking algorithm settings, or adjusting how the experiment is set up. In almost every case, the answer was no: circuits changed dramatically, and two researchers studying the same model on the same task could easily reach completely different conclusions. We traced this back to a fundamental issue: the measurements used to identify important connections in the network are noisy, and common shortcuts used to make the analysis faster make this worse. We provide concrete recommendations for how the field can do better, borrowing standard practices from statistics such as repeating analyses on resampled data and reporting how stable the results are. Our findings suggest that many published circuits should be treated with more caution than they currently are.