Query Circuits: Explaining How Language Models Answer User Prompts

Explaining why a language model produces a particular output requires local, input-level explanations. Existing methods uncover global capability circuits (e.g., indirect object identification), but not why the model answers a specific input query in a particular way. We introduce query circuits, which directly trace the information flow inside a model that maps a specific input to the output. Unlike surrogate-based approaches (e.g., sparse autoencoders), query circuits are identified within the model itself, resulting in more faithful and computationally accessible explanations. To make query circuits practical, we address two challenges. First, we introduce Normalized Deviation Faithfulness (NDF), a robust metric to evaluate how well a discovered circuit recovers the model's decision for a specific input, and is broadly applicable to circuit discovery beyond our setting. Second, we develop sampling-based methods to efficiently identify circuits that are sparse yet faithfully describe the model’s behavior. Across benchmarks (IOI, arithmetic, MMLU, and ARC), we find that there exist sparse query circuits within the model that recover much of its performance on single queries. For example, on average, a circuit covering only 1.3\% of model connections can recover about 60\% of performance on an MMLU question. Overall, query circuits provide a step towards faithful, scalable explanations of how language models process individual inputs.

Large language models make consequential decisions, yet we often do not know how they arrive at specific answers. To deploy them safely in high-stakes domains such as medicine or autonomous driving, we need methods that explain individual model decisions. Prior work has primarily focused on explaining models' general capabilities. Although recent studies have explored input-specific explanations, they often rely on simplified copies (surrogate models) that may not faithfully reflect the original model’s computation. We instead introduce query circuits: compact sub-networks identified directly within the original model that drive its response to a specific query. We formalize the task of query circuit discovery, identify its key technical challenges, and propose both a more reliable evaluation metric and a sampling-based discovery method. Across diverse benchmarks, we find that compact circuits within the model can explain much of its behavior on a given query. On the multiple-choice benchmark MMLU, query circuits containing only 1.3% of the model’s connections can, on average, recover roughly 60% of the model’s performance on a given question. When combined with existing interpretability tools, these circuits can be made human-interpretable, providing a practical path toward explaining individual LLM decisions.