Do Activation Verbalization Methods Convey Privileged Information?

Recent interpretability methods have proposed to translate LLM internal representations into natural language descriptions using a second verbalizer LLM. This is intended to illuminate how the target model represents and operates on inputs. But do such activation verbalization approaches actually provide privileged knowledge about the internal workings of the target model, or do they merely convey information about the inputs provided to it? We critically evaluate popular verbalization methods and datasets used in prior work and find that one can perform well on such benchmarks without access to target model internals, suggesting that these datasets are not ideal for evaluating verbalization methods. We then run controlled experiments which reveal that verbalizations often reflect the parametric knowledge of the verbalizer LLM that generated them, rather than the knowledge of the target LLM whose activations are decoded. Taken together, our results indicate a need for targeted benchmarks and experimental controls to rigorously assess whether verbalization methods provide meaningful insights into the operations of LLMs.

How do we know whether the information that we extract from the hidden representations of a language really represents the language model’s “thoughts”? We wanted to see whether these “thoughts”, or what we call “privileged knowledge”, might be obtainable by stress testing these language models, called activation verbalizers, which read out the contents of a target model’s activations. To do this, we took an activation and tried to invert its contents, and we also constructed controlled datasets where we train our target model (that we inspect with our verbalizer, which has not been trained on these datasets). We find that for many existing tasks that purport to evaluate verbalization performance, we can often achieve very similar results by simply reconstructing a model's activations back into text, which means we might not be accessing any “privileged knowledge”. We also find that, when tested on a dataset that focuses on “privileged knowledge”, existing verbalization methods fail. Our findings have implications on how future researchers design evaluations for activation verbalizers.