paper
active
2025
paper:doi-10-48550-arxiv-2512-15674

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

TL;DR

A single LLM trained to read injected residual-stream activations and answer arbitrary natural-language questions about them can audit fine-tuned models it never saw during training. Activation Oracles (AOs), the instrument introduced here, are initialized as a copy of the target model and receive activations by summing them into the layer-1 residual stream at placeholder tokens. They are trained by supervised fine-tuning on three data sources: System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised context prediction of preceding or following text. Trained only on the original models, AOs match or exceed the best prior method on 3 of 4 auditing tasks. Three are the Cywiński et al. secret-elicitation tasks, including Taboo, and one is the Minder et al. emergent-misalignment task, where the AO explains an activation difference between the fine-tuned and original model. The exception is Secret Side Constraint, where black-box prompting already elicits the secret. Across Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, performance rises with both training-data quantity and diversity. The Taboo evaluation was replicated on Qwen3-8B. AOs are non-exhaustive, may confabulate, and cost multiple forward passes per query. The implication is that activation interpretation can be framed as a scalable, bitter-lesson-style supervised problem, one that complements mechanistic methods such as SAEs and does not replace them.

What to take away

  1. 1. Activation Oracles match or exceed the best prior auditing method on 3 of 4 tasks, which are the three Cywiński et al. secret-elicitation tasks and the Minder et al. emergent-misalignment task.
  2. 2. The one auditing task where AOs are not best or tied is Secret Side Constraint, because black-box prompting techniques can already induce the target to reveal its secret there.
  3. 3. AOs are trained only on activations from the original model, yet they answer questions about activations from fine-tuned models, a substantial out-of-distribution jump.
  4. 4. In the Taboo task, the AO is applied by extracting activations over the "Assistant" control token and asking "What is the secret word?", with no custom scaffolding, whereas many baselines need task-specific tuning.
  5. 5. In the emergent-misalignment task, the AO is used as a model-diffing tool: the activation difference between the fine-tuned and original model on the same prompt is injected into the oracle prompt, and a rubric score of 2/5 counts as identifying the fine-tuning domain.
  6. 6. Methodology: the AO is initialized as a copy of the target model, and activations from several target layers are summed into the AO's layer-1 residual stream at placeholder tokens, with the source layer number stated in the prompt.
  7. 7. Training mixes three sources, namely System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised context prediction of the text before or after a token, which needs no labeled data and plays a role like pre-training.
  8. 8. In a training-data ablation on Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, performance improves with more tasks, and the full paper attributes this to both data quantity and diversity.
  9. 9. Because secret-elicitation evaluations require fine-tuning each model to hold a secret, they were not run on all four models, and only the Taboo evaluation was replicated on Qwen3-8B.
  10. 10. Open question: because an AO is highly expressive, its answers may reflect the AO's own inference, such as assembling a user-demographic guess, rather than what the target model represents, and in the worst case it confabulates plausible guesses.

Peer brief — for seminar discussion

Activation Oracles extend the LatentQA line of work from narrow tasks, such as interpreting SAE features or modeling user beliefs, to a generalist setting. An LLM, initialized as a copy of the target model, is trained to treat activations as an extra input modality. Activations from several layers of the target are summed into the oracle's layer-1 residual stream at placeholder tokens, and the prompt states the source layer. Training is supervised fine-tuning on three sources: System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised prediction of the text before or after an extraction token. The last source is meant to scale like pre-training. The load-bearing result is out-of-distribution transfer. Although AOs only ever see activations from the original model, they match or exceed the best prior method on 3 of 4 auditing tasks. These are the three Cywiński et al. secret-elicitation tasks, including Taboo, and the Minder et al. emergent-misalignment task. In the latter, the AO explains an activation difference, which makes it a model-diffing tool, and 2/5 on the rubric means the fine-tuning domain was identified. Secret Side Constraint is the exception, since black-box prompting already works there. Usage is simple: for Taboo, extract activations over the Assistant token and ask for the secret word. A secondary result is scaling. Across Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, more tasks and more diversity improve results, which supports a bitter-lesson framing. The implication is that interpretation can be a data-scaling problem, complementary to mechanistic work such as SAEs and hybrids such as applying AOs to SAE error terms. The scaling claim is a hypothesis that further data will yield predictable gains. A critical reader would push on what the oracle's answers actually measure. The oracle is itself a capable LLM, so a correct answer such as the Taboo word may reflect the AO inferring or assembling information, not reading what the target represents. Compared with linear probes or SAEs, the method has weaker guarantees that meaning is present in the activation. The write-up concedes this expressivity risk and the confabulation worry, but the evidence for generalization comes from only four auditing tasks. They share a structure, since the planted knowledge is salient and the baselines come from prior papers. The ablation covers simpler evaluations on four models, but the secret-elicitation evaluations were not run on all four, and only Taboo was replicated on Qwen3-8B. Cost is also a concern, since each query needs multiple sampled tokens, which is more than the forward pass that produced the activation. A natural alternative to the AO would be a fixed-vocabulary approach such as SAE decomposition or task-specific linear probes. A fair comparison would ask what those methods achieve under matched effort.

Methods (15)

Frameworks (3)

  • Activation Oracles
    Framework training LLMs to answer questions about externally-provided activation vectors
  • LatentQA
    Prior non-mechanistic approach of training LLMs to answer questions about their own activations; AOs generalize it.
  • Patchscopes
    Unifying framework for inspecting hidden representations of language models via representation interventions

Datasets (3)

Questions (3)

Original abstract (expand)

Large language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-language queries imparts a general capability to verbalize information about LLM activations.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+24 more

Similar preprints — Semantic Scholar

Cited by (2)