Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Large language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-language queries imparts a general capability to verbalize information about LLM activations.

Modern AI systems process information through billions of internal signals that are extremely difficult for humans to interpret. Understanding these internal signals is important for safety, since an AI model could develop hidden behaviors or goals that are not apparent from its outputs alone. Existing techniques for inspecting these signals require deep technical expertise and custom tooling for each new question a researcher wants to ask. We trained AI models to answer plain-language questions about what is happening inside other AI models. For example, given the internal signals from a model that was secretly trained to manipulate users, we can simply ask "What is this model's hidden goal?" and receive a natural-language answer. We call these question-answering models Activation Oracles. We found that Activation Oracles can uncover hidden information that was never stated in the text the model was processing, such as secret words or concealed objectives. They matched or outperformed specialized inspection tools on most of the tasks we tested, despite requiring no task-specific setup. This approach could make AI safety auditing more accessible by letting practitioners ask questions in plain language rather than requiring specialized interpretability expertise.