Local Causal Attribution of Chain-of-Thought Reasoning

Understanding the causal structure of a language model's thought process is a problem of significant importance for both transparency and safety. In this work, we take a step toward this goal by analyzing the causal relationships among individual components, termed units, of a given chain-of-thought trace. Specifically, we construct a structural causal model on these units and relate each unit to the log probability of generating (subsequent) output units. Our algorithm, termed AttriCoT, is a black-box method that performs attribution by estimating importance parameters in the structural causal model using $O(U)$ forward passes through the model, where $U$ is the number of units. Evaluation of perturbation curves across 5 datasets and 4 reasoning models shows that AttriCoT produces attributions that are more faithful to the model's behavior than alternative methods. The attribution results also reveal notable differences in thought structure between models and domains.