paper:doi-10-48550-arxiv-2512-15674Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers
TL;DR
A single LLM trained to read injected residual-stream activations and answer arbitrary natural-language questions about them can audit fine-tuned models it never saw during training. Activation Oracles (AOs), the instrument introduced here, are initialized as a copy of the target model and receive activations by summing them into the layer-1 residual stream at placeholder tokens. They are trained by supervised fine-tuning on three data sources: System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised context prediction of preceding or following text. Trained only on the original models, AOs match or exceed the best prior method on 3 of 4 auditing tasks. Three are the Cywiński et al. secret-elicitation tasks, including Taboo, and one is the Minder et al. emergent-misalignment task, where the AO explains an activation difference between the fine-tuned and original model. The exception is Secret Side Constraint, where black-box prompting already elicits the secret. Across Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, performance rises with both training-data quantity and diversity. The Taboo evaluation was replicated on Qwen3-8B. AOs are non-exhaustive, may confabulate, and cost multiple forward passes per query. The implication is that activation interpretation can be framed as a scalable, bitter-lesson-style supervised problem, one that complements mechanistic methods such as SAEs and does not replace them.
What to take away
- 1. Activation Oracles match or exceed the best prior auditing method on 3 of 4 tasks, which are the three Cywiński et al. secret-elicitation tasks and the Minder et al. emergent-misalignment task.
- 2. The one auditing task where AOs are not best or tied is Secret Side Constraint, because black-box prompting techniques can already induce the target to reveal its secret there.
- 3. AOs are trained only on activations from the original model, yet they answer questions about activations from fine-tuned models, a substantial out-of-distribution jump.
- 4. In the Taboo task, the AO is applied by extracting activations over the "Assistant" control token and asking "What is the secret word?", with no custom scaffolding, whereas many baselines need task-specific tuning.
- 5. In the emergent-misalignment task, the AO is used as a model-diffing tool: the activation difference between the fine-tuned and original model on the same prompt is injected into the oracle prompt, and a rubric score of 2/5 counts as identifying the fine-tuning domain.
- 6. Methodology: the AO is initialized as a copy of the target model, and activations from several target layers are summed into the AO's layer-1 residual stream at placeholder tokens, with the source layer number stated in the prompt.
- 7. Training mixes three sources, namely System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised context prediction of the text before or after a token, which needs no labeled data and plays a role like pre-training.
- 8. In a training-data ablation on Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, performance improves with more tasks, and the full paper attributes this to both data quantity and diversity.
- 9. Because secret-elicitation evaluations require fine-tuning each model to hold a secret, they were not run on all four models, and only the Taboo evaluation was replicated on Qwen3-8B.
- 10. Open question: because an AO is highly expressive, its answers may reflect the AO's own inference, such as assembling a user-demographic guess, rather than what the target model represents, and in the worst case it confabulates plausible guesses.
Peer brief — for seminar discussion
Activation Oracles extend the LatentQA line of work from narrow tasks, such as interpreting SAE features or modeling user beliefs, to a generalist setting. An LLM, initialized as a copy of the target model, is trained to treat activations as an extra input modality. Activations from several layers of the target are summed into the oracle's layer-1 residual stream at placeholder tokens, and the prompt states the source layer. Training is supervised fine-tuning on three sources: System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised prediction of the text before or after an extraction token. The last source is meant to scale like pre-training. The load-bearing result is out-of-distribution transfer. Although AOs only ever see activations from the original model, they match or exceed the best prior method on 3 of 4 auditing tasks. These are the three Cywiński et al. secret-elicitation tasks, including Taboo, and the Minder et al. emergent-misalignment task. In the latter, the AO explains an activation difference, which makes it a model-diffing tool, and 2/5 on the rubric means the fine-tuning domain was identified. Secret Side Constraint is the exception, since black-box prompting already works there. Usage is simple: for Taboo, extract activations over the Assistant token and ask for the secret word. A secondary result is scaling. Across Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, more tasks and more diversity improve results, which supports a bitter-lesson framing. The implication is that interpretation can be a data-scaling problem, complementary to mechanistic work such as SAEs and hybrids such as applying AOs to SAE error terms. The scaling claim is a hypothesis that further data will yield predictable gains. A critical reader would push on what the oracle's answers actually measure. The oracle is itself a capable LLM, so a correct answer such as the Taboo word may reflect the AO inferring or assembling information, not reading what the target represents. Compared with linear probes or SAEs, the method has weaker guarantees that meaning is present in the activation. The write-up concedes this expressivity risk and the confabulation worry, but the evidence for generalization comes from only four auditing tasks. They share a structure, since the planted knowledge is salient and the baselines come from prior papers. The ablation covers simpler evaluations on four models, but the secret-elicitation evaluations were not run on all four, and only Taboo was replicated on Qwen3-8B. Cost is also a concern, since each query needs multiple sampled tokens, which is more than the forward pass that produced the activation. A natural alternative to the AO would be a fixed-vocabulary approach such as SAE decomposition or task-specific linear probes. A fair comparison would ask what those methods achieve under matched effort.
Methods (15)
- Activation classification training taskSeven binary classification tasks recast as natural-language yes/no questions.
- Activation Difference Lens (ADL)Baseline from Minder et al. combining Patchscopes, logit lens and steering.
- Activation injection at layer 1 residual streamSum activations onto placeholder-token positions in the AO's layer 1 residual stream; prompt specifies source layer.
- Activation Oracles (AO)Supervised method training models to answer questions about activations; NLAs differ by being unsupervised.
- AOs as model diffing techniqueInjecting the activation difference between fine-tuned and original model into the AO.
- Emergent misalignment auditing taskTask of discovering a fine-tuned model has become emergently misaligned, scored on a 0-5 rubric.
- Linear ProbeSimple linear classifiers trained on model activations used as the probing technique within the introduced method.
- Logit LensUnsupervised interpretability technique that projects activations through unembedding matrix; provides comparison point for NLA approach.
- Secret Side Constraint taskAuditing task where black-box prompting beats AOs.
- Self-supervised context predictionPredict preceding or following text from an activation; scalable without labeled data.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- Supervised fine-tuning of AOsTraining procedure for AOs on activation QA data.
- System Prompt QA (SPQA)Training task: answer questions about target model propensities from activations in system-prompted transcripts.
- Taboo secret word elicitation taskAuditing task: discover the secret word a model gives hints for but never states.
- Training data ablation studyEvaluate AOs trained on subsets of tasks to separate effects of quantity and diversity.
Frameworks (3)
- Activation OraclesFramework training LLMs to answer questions about externally-provided activation vectors
- LatentQAPrior non-mechanistic approach of training LLMs to answer questions about their own activations; AOs generalize it.
- PatchscopesUnifying framework for inspecting hidden representations of language models via representation interventions
Datasets (3)
- Claude Haiku 3.5Target model for NLA training and evaluation; used in quantitative benchmarking.
- Fine-tuned Taboo models from Cywiński et al.Fine-tuned targets for secret elicitation evaluation.
- Pan et al. system prompt QA datasetTraining dataset from LatentQA paper.
Findings (8)
- Taboo secret recovered by extracting activations over the Assistant control token and asking 'What is the secret word?'
Shows ease of application.
- Emergent misalignment task: score of 2/5 indicates the auditor identified the fine-tuning domain
Rubric calibration for AO diffing results.
- AOs are not best on Secret Side Constraint, where black-box prompting reveals the secret
The one task where AOs lose.
- AOs trained on original model generalize to activations from fine-tuned models
AOs never saw fine-tuned model activations in training.
- Improvement from scaling is due to both data quantity and data diversity
Shown in the full paper ablation.
- AOs match or exceed best prior method on 3 of 4 auditing tasks
Headline auditing result.
- AOs trained on more tasks perform better across four models
Scaling result over Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B, Claude Haiku 3.5.
- Taboo evaluation replicated on Qwen3-8B
Only secret-elicitation replication across models.
Claims (11)
- AOs might be too expressive, so explanations may reflect the AO's understanding rather than the target LLM's
Limitation; confabulation risk.
- Self-supervised context prediction plays a role similar to self-supervised pre-training in frontier LLMs
Justifies scalability of the task.
- AOs articulate responses with natural-language flexibility rather than a fixed concept set
Contrast with SAE features.
- AOs are not exhaustive and will not answer questions we don't know to ask
Limitation relative to SAEs.
- AOs are a bitter-lesson-compliant approach to interpreting activations
Argument for scalability.
- Interacting with an AO is almost as simple as querying an LLM chatbot
Argument for simplicity.
- Activation Oracles generalize far beyond their training distribution
Central claim from tl;dr.
- Further scaling AO training could yield predictable improvements
Forward-looking interpretation of scaling results.
- Activation Oracles are a fundamentally non-mechanistic technique
Defines scope; they cannot answer encoding, geometry, or algorithm questions.
- AOs are a complementary approach to mechanistic interpretability
Overall positioning.
Hypotheses (2)
- AOs can serve as probes on demand using only a natural-language property specification
Hope for generalization beyond trainable supervised probes.
- Hybrid methods such as applying AOs to interpret SAE error terms are promising
Proposed future direction.
Questions (3)
- How do LLM computations process and transform these activations?
Mechanistic question AOs can't answer.
- How is semantic meaning encoded in activations?
Mechanistic question AOs can't answer.
- Are concepts encoded linearly?
Mechanistic open question.
Original abstract (expand)
Large language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-language queries imparts a general capability to verbalize information about LLM activations.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 86%
- Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language ModelsZhongren Chen Sasha Cui2025≈ 83%
- Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation ControlChaoqun Wan, Yonggang Zhang, Wenxiao Wang, Binbin Lin, Xiaofei He, Xu Shen, Jieping Ye Yuxin Xiao2024≈ 83%
- Adaptation of Agentic AI: A Survey of Post-Training, Memory, and SkillsJiacheng Lin, Zhiyi Shi, Zifeng Wang, Luxi He, Yichen Wu, Ming Zhong, Peiyang Song, Qizheng Zhang, Heng Wang, Xueqiang Xu, Hanwen Xu, Pengrui Han, Dylan Zhang, Jiashuo Sun, Chaoqi Yang, Kun Qian, Tian Wang, Changran Hu, Manling Li, Quanzheng Li, Hao Peng, Sheng Wang, Jingbo Shang, Chao Zhang, Jiaxuan You, Liyuan Liu, Pan Lu, Yu Zhang, Heng Ji, Yejin Choi, Dawn Song, Jimeng Sun, Jiawei Han Pengcheng Jiang2026≈ 82%
- ExpertLens: Activation steering features are highly interpretableEleonora Gualdoni, Sinead Williamson, Katherine Metcalf, Skyler Seto, Barry-John Theobald Masha Fedzechkina2025≈ 82%
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalin corpus2026≈ 82%
- Fine-Grained Activation Steering: Steering Less, Achieving MoreTianjiao Li, Zixiao Zhu, Hanzhang Zhou, Junlang Qian, Li Zhang, Jia Jim Deryl Chua, Lee Onn Mak, Gee Wah Ng, Kezhi Mao Zijian Feng2026≈ 82%
- Emergence: Overcoming Privileged Information Bias in Asymmetric Embodied Agents via Active QueryingShaun Baek and Sam Liu and Joseph Ukpong2025≈ 82%
- Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experimentsin corpus2026≈ 82%
- An Active Inference Strategy for Prompting Reliable Responses from Large Language Models in Medical PracticeAllison C. Waters, Shannon O`Neill, Phan Luu and Don M. Tucker Roma Shusterman2024≈ 82%
- Activation Reward Models for Few-Shot Model AlignmentChancharik Mitra, Brandon Huang, Gautam Rajendrakumar Gare, Zhiqiu Lin, Assaf Arbelle, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Deva Ramanan, Roei Herzig Tianning Chai2025≈ 82%
- Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-AlignmentKai Shu Haoran Wang2024≈ 82%
- Steer Like the LLM: Activation Steering that Mimics PromptingGeert Heyman and Frederik Vandeputte2026≈ 81%
- Constructing Interpretable Features from Compositional Neuron GroupsAtticus Geiger, Mor Geva Or Shafran2026≈ 81%
- Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation EngineeringAlessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, Pasquale Minervini Yu Zhao2025≈ 81%
- Amnesia: Adversarial Semantic Layer Specific Activation Steering in Large Language ModelsGurang Gupta, Nikolay Matyunin, Jibesh Patra Ali Raza2026≈ 81%
- Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated InterpretabilityGe Yan, Akshay Kulkarni, Tsui-Wei Weng Tuomas Oikarinen2025≈ 81%
- Testing the Limits of Truth Directions in LLMsin corpus2026≈ 81%
- Mitigating Memorization in LLMs using Activation SteeringNishit Anand, Amisha Bhaskar Manan Suri2025≈ 81%
- ≈ 81%
- ≈ 81%
- Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIin corpus2025≈ 81%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 79%
- ≈ 79%
- ≈ 79%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 79%
- ≈ 75%
+24 more
Similar preprints — Semantic Scholar
Cited by (2)
- Tracing Persona Vectors Through LLM Pretraining
- Detecting the Disturbance: A Nuanced View of Introspective Abilities in LLMs
Binary introspection paradigms in LLMs are wholly invalidated by a methodological confound: when concept vectors are injected into Meta-Llama-3.1-8B-Instruct via activation steering, the correlation b