Query Lens: Interpreting Sparse Key-Value Features with Indirect Effects

While sparse autoencoders provide features more interpretable than individual neurons, reliably characterizing them remains challenging. We propose Query Lens, which extends Logit Lens to enable more comprehensive and faithful interpretations of sparse features. By jointly considering encoder-side key features and decoder-side value features, we identify both the inputs that activate a feature and the outputs it promotes. We also account for indirect, module-mediated effects that arise when the feature is processed by downstream modules, going beyond the direct effect captured by Logit Lens. In experiments, we find that Query Lens yields coherent token signatures for features that remain uninterpretable under Logit Lens. Finally, we propose the Subspace Channel Hypothesis, suggesting that downstream modules read features through layer-specific subspaces.

Modern AI language models run on a tangle of numbers that's nearly impossible to read directly. To make sense of it, researchers isolate small internal "features" and try to label what each one does. The usual shortcut translates a feature straight into the words it favors. That often returns meaningless fragments, and even when it works, it only tells you what the feature pushes the model to say, not what makes it fire in the first place. Query Lens addresses both gaps. It reads each feature from both ends: the words that switch it on, and the words it promotes. And rather than stopping at the feature's immediate effect, it follows what happens downstream, as later layers pick the feature up and carry it forward. With those two changes, features that used to read like gibberish resolve into single, coherent themes. One that previously decoded to random symbols turns out to be about food. This makes it easier to say what a model has actually learned, and you don't have to comb through huge amounts of text to do it. One result surprised us: each part of the model seems to listen to features through its own dedicated "channel." That tells us something about how information moves inside these systems, and it gives a concrete handle for inspecting or adjusting a model's behavior more safely.