thinker:evans-owainEvans, Owain
Authored papers (2)
A single LLM trained to read injected residual-stream activations and answer arbitrary natural-language questions about them can audit fine-tuned models it never saw during training. Activation Oracles (AOs), the instrument introduced here, are initialized as a copy of the target model and receive activations by summing them into the layer-1 residual stream at placeholder tokens. They are trained by supervised fine-tuning on three data sources: System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised context prediction of preceding or following text. Trained only on the original models, AOs match or exceed the best prior method on 3 of 4 auditing tasks. Three are the Cywiński et al. secret-elicitation tasks, including Taboo, and one is the Minder et al. emergent-misalignment task, where the AO explains an activation difference between the fine-tuned and original model. The exception is Secret Side Constraint, where black-box prompting already elicits the secret. Across Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, performance rises with both training-data quantity and diversity. The Taboo evaluation was replicated on Qwen3-8B. AOs are non-exhaustive, may confabulate, and cost multiple forward passes per query. The implication is that activation interpretation can be framed as a scalable, bitter-lesson-style supervised problem, one that complements mechanistic methods such as SAEs and does not replace them.
Finetuning-induced personality shifts in LLMs—including unintended ones—are strongly predicted and controllable by linear directions in activation space called persona vectors. On Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, the magnitude of activation shift along a persona vector after finetuning correlates with post-finetuning trait expression at r = 0.76–0.97 across traits including evil, sycophancy, and hallucination—substantially above cross-trait baselines of r = 0.34–0.86. An automated pipeline using Claude 3.7 Sonnet to generate contrastive system prompts and GPT-4.1-mini as a judge extracts these vectors from any natural-language trait description without bespoke data curation. Beyond post-hoc inhibition, a novel preventative steering method—amplifying the undesired persona direction during finetuning to cancel gradient pressure—limits trait acquisition while better preserving MMLU accuracy than inference-time steering; multi-layer preventative steering suppresses traits to near-baseline levels even on intentionally trait-eliciting datasets. At the data level, a projection difference metric—comparing training response projections onto the persona vector against base model natural response projections—predicts post-finetuning trait expression before any training occurs and identifies problematic samples in LMSYS-CHAT-1M that evade LLM-based filtering, including underspecified prompts that induce hallucination without explicit fabrication. This paper argues these findings imply that model personas are latent factors encoded linearly in residual stream activations, that cross-domain misalignment generalizes through these directions, and that proactive persona-vector monitoring and preventative steering should be standard components of responsible finetuning pipelines.
More papers — OpenAlex / S2
Co-authors (12)
- Arditi, Andy4 shared
- Chen, Runjin4 shared
- Lindsey, Jack4 shared
- Sleight, Henry4 shared
- Adam Karvonen2 shared
- Andy Arditi2 shared
- Henry Sleight2 shared
- Jack Lindsey2 shared
- Runjin Chen2 shared
- Arnab Sen Sharma1 shared
- Chua, James1 shared
- Clément Dumas1 shared
Their work is cited by (12)
- Tracing Persona Vectors Through LLM Pretraining3× refs
- Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI2× refs
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders2× refs
- Facet-Level Persona Control by Trait-Activated Routing with Contrastive SAE for Role-Playing LLMs2× refs
- Steering at the Source: Style Modulation Heads for Robust Persona Control2× refs
- Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs2× refs
- Where is the Mind? Persona Vectors and LLM Individuation2× refs
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior2× refs
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Training2× refs
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering2× refs
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution2× refs
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectors2× refs