Current Activation Oracles Are Hard to Use on Safety-Relevant Tasks

Activation oracles (AOs) are LLMs finetuned to answer questions about another model’s activations (Karvonen et al., 2025). We test the publicly released Qwen3 AOs on out-of-distribution safety-relevant tasks and find them difficult to use off the shelf. AO responses are often vague to the point of being unfalsifiable, and confidently hallucinate when pushed to be specific. Many apparent successes are explained by a confound we call text inversion: because most of the AO’s training involves predicting tokens near a given activation, AOs can answer by paraphrasing nearby decoded text rather than reading deeper internal state. A related confound is that the AO is itself a capable LLM, and can sometimes answer from its own weights without using the activations at all. We ran three tasks where text inversion cannot help, and AOs were at or near chance on all three. Given an arithmetic problem and asked to predict the answer from activations before any answer tokens, the AO emits the same handful of numbers regardless of the problem. Given a chain of thought that rationalizes a user-preference-flipped answer without ever mentioning the user, the AO cannot reliably tell it apart from a non-sycophantic rollout. Given a logic puzzle missing key information, the AO cannot identify what the model is confused about. When tasked with identifying why a reasoning model backtracks, the AO does better, but a follow-up edit experiment suggests its correct answers mostly restate keywords near the probe. As an additional case study, we apply AOs to censored topics in Chinese models and find the oracle’s own pretraining knowledge dominates, illustrating the second confound. AOs do show signal in narrower settings closer to their training distribution: detecting subtle activation steering before it surfaces in text, multiple-choice selection among plausible backtracking reasons, and next/previous-token prediction. We read this as a meaningful negative update on current released AOs as general-purpose safety tools, though the limitations look fixable. Text-inversion-controlled tasks and a no-activation ablation should become standard checks for future AO evaluations.