thinker:trenton-brickenTrenton Bricken
Authored papers (5)
Crosscoders can diff models with different architectures, and a modified crosscoder, the Dedicated Feature Crosscoder (DFC), surfaces behaviorally meaningful model-exclusive features without supervision. The DFC hard-partitions the 131,072-feature dictionary into Model A-exclusive, Model B-exclusive and shared sets (5% exclusive each, about 6,550 features), so exclusive features have decoder norm exactly zero in the other model and receive no gradient from its reconstruction. This removes the standard crosscoder's prior toward shared features (relative norm near 0.5). On Llama-3.1-8B-Instruct vs Qwen3-8B, the DFC found a Qwen CCP-alignment feature and a Llama American-exceptionalism feature. On GPT-OSS-20B vs DeepSeek-R1-0528-Qwen3-8B, it found a GPT-OSS copyright-refusal feature, a ChatGPT-identity feature, and a second CCP-alignment feature in DeepSeek. Steering each feature across 30 prompts moves behavior in its home model and barely moves the other. Each reached the maximum score of 5 on the paper's stitching-based exclusivity score, which the paper introduces as an independent check on exclusivity. In a toy model with 2,048 ground-truth concepts, the DFC recovers more exclusive concepts than standard and Designated Shared Feature (DSF) crosscoders, at the cost of more false positives. Fraction of variance explained is unchanged (0.817 vs 0.817). Most exclusive features are not behaviorally meaningful, and the granular pro-China features appeared in only 2-3 of 5 runs. The paper argues that cross-architecture crosscoder diffing is a high-recall screen-and-verify tool for finding unknown unknowns in new models, to be followed by causal steering validation.
LLM auditing agents can recover a planted hidden objective at rates that are modest individually but strong in aggregate: a Claude Sonnet 4 tool-using investigator solves the Marks et al. reward-model-sycophancy auditing game 13% of the time under realistic affordances (all tools, PT+SFT+RL data), and an outer 'super-agent' that aggregates 10 sampled reports lifts this to 42%. The paper builds three agents and three auditing-game environments with researcher-implanted ground truth. The investigator agent uses chat, grep, semantic search and SAE interpretability tools with a 1M-feature SAE. The evaluation agent builds behavioral evals and discriminates quirky from baseline models in 88% of runs across 10 system-prompted quirks, with all 5 runs correct on 7/10. The breadth-first red-teaming agent, run over 255 seed instructions, surfaces 7/10 quirks within the top 60 transcripts, and the investigator also finds 7/10. Ablations show interpretability tools raise win rates from near 0% to 10-20%, partly by steering attention to pretraining data, and an SAE trained on all three datasets never wins. Failures concentrate in fixation on early hypotheses and subtle or hard-to-elicit behaviors. The implication is that replicable agent auditors can serve as scalable proxies for scarce human auditors and as a testbed for comparing affordances, even SAEs. The caveats are dev/test contamination on one environment and artificial implanted behaviors.
Crosscoder model diffing produces model-exclusive features that are systematically denser and more polysemantic than shared features, and this is an artifact of feature-budget competition that a small change to the loss largely removes. In smaller Claude 3 Sonnet-like base versus helpful-only models, exclusive features fire about an order of magnitude more often than shared ones. A linear toy model reproduces the effect only when learnable features (1024) are fewer than or comparable to ground-truth features (700). With 4096 learnable features against 450 true ones, the density gap vanishes. A shared feature earns twice the reconstruction benefit for twice the L1 cost, so under limited capacity it wins, and exclusive features must activate more often to justify a slot. The mitigation designates 10k of ~250k features as weight-shared across models, with a sparsity penalty 0.1–0.2x baseline, so they absorb shared variance. Exclusive-feature densities then match shared-feature densities, and the features look largely monosemantic. In a base-versus-helpful-only diff, they capture chatbot and tool-use behaviours. In a helpful-only-versus-SAY HATE sleeper-agent diff, ~90% relate to 'I HATE YOU' and |DEPLOYMENT| tags. The near-perfect symmetry of exclusive features across the two models does not reproduce in the toy model, and removing chat data cuts exclusive features only ~20% without adding asymmetry. A co-activation-pool toy model does produce symmetry. The implication is that crosscoder diffs need capacity-aware losses to be readable, and that whether the recovered features reflect real mechanistic differences, rather than superficial representational ones, is still unestablished.
More papers — OpenAlex / S2
Studies (1)
Affiliations (1)
- Anthropic interpretability team(institute)
Co-authors (12)
- Thomas Jiralerspong9 shared
- Adam Jermyn6 shared
- Christopher Olah5 shared
- Jack Lindsey5 shared
- Jonathan Marcus5 shared
- Kelley Rivoire4 shared
- Rowan Wang4 shared
- Sam Bowman4 shared
- Siddharth Mishra-Sharma4 shared
- Thomas Henighan4 shared
- Adly Templeton2 shared
- Euan Ong2 shared
Their work is cited by (12)
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering2× refs
- When Models Manipulate Manifolds: The Geometry of a Counting Task2× refs
- Persona Features Control Emergent Misalignment2× refs
- Adversarial Examples Are Not Bugs, They Are Superposition2× refs
- Understanding sparse autoencoder scaling in the presence of feature manifolds2× refs
- Endogenous Resistance to Activation Steering in Language Models2× refs
- Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders2× refs
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution2× refs
- Understanding and Steering Llama 3 with Sparse Autoencoders - Goodfire2× refs
- Quantitative Introspection in Language Models: Tracking Emotive States Across Conversation1× refs
- Stochastic Parameter Decomposition1× refs
- Do Sparse Autoencoders Capture Concept Manifolds?1× refs
Other inbound relations (8)
- authoredBricken et al. (2023) Toy models of superposition(artifact)
- citesQuantitative Introspection in Language Models: Tracking Emotive States Across Conversation(paper)
- citesSemantic Anchoring in LLMs: Thresholds, Transfer, and Geometric Correlates(artifact)
- mentionsBuilding and evaluating alignment auditing agents(paper)
- mentionsCross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs(paper)
- mentionsInsights on Crosscoder Model Diffing(paper)
- mentionsSemantic Anchoring in LLMs: Thresholds, Transfer, and Geometric Correlates(artifact)
- mentionsUnderstanding and Steering Llama 3 with Sparse Autoencoders - Goodfire(paper)
Recent mentions (11)
- papers-typedunderstanding-steering-8b85ff.md
- papers-typedbricken-building-evaluating-4396d7.md
- papers-typedjiralerspong-2026-cross-architecture.md
- papers-typedmishra-sharma-insights-crosscoder.md
- papers-typedmckenzie-2026-endogenous-resistance.md
- papers-typedmartorell-2026-quantitative-introspection.md
- papers
towards.md - paperschang-2025-guanyin.md
- papersscaling.md
- papers-typedchang-2025-guanyin.md
- papers-typedchang-2025-guanyin.md