thinker:neel-nandaNeel Nanda
Authored papers (8)
Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.
Gains on the sparsity-fidelity frontier do not predict how well a sparse autoencoder performs on downstream tasks, and SAEBench demonstrates this by scoring over 200 SAEs on eight metrics. Matryoshka BatchTopK SAEs rank below TopK and BatchTopK on loss recovered, yet they take the best score on Absorption, RAVEL, Sparse Probing, SCR and TPP, which is 5 of 8 metrics, in the typical L0 range of 40-200. Matryoshka is also the only architecture whose disentanglement scores improve when width grows from 4k to 16k to 65k latents on Gemma-2-2B (layer 12). The other architectures show inverse scaling on absorption and SCR, and the effect persists when the number of ablated latents is held fixed. SAEBench combines four capability groups (concept detection, interpretability, reconstruction, feature disentanglement). It adds two novel zero-ablation metrics, Spurious Correlation Removal and Targeted Probe Perturbation, and it extends the absorption metric to partial and shared absorption. The suite covers seven architectures (ReLU, TopK, BatchTopK, Gated, JumpReLU, P-annealing, Matryoshka), Gemma-2-2B and Pythia-160M, and Gemma-Scope SAEs up to 1M latents. Hierarchical architectures beat the rest by 30-40% on SCR and absorption. No single L0 wins everywhere, though L0 of 50-150 is a reasonable compromise. The implication is that SAE progress needs a multi-metric benchmark, and that optimizing reconstruction at fixed sparsity can hide failures in concept isolation, which grow with scale. The paper declines to combine the metrics into one score.
Contrastive activation steering can suppress evaluation-awareness and elicit genuine deployment behavior from a deliberately trained model organism, not merely silence verbalizations of being tested. Working with Llama 3.3 Nemotron Super 49B, the authors trained an evaluation-aware model using a two-step pipeline—synthetic document fine-tuning (SDF) on 115.6 million tokens followed by four rounds of expert iteration on 41,290 examples—that causes the model to write Python type hints 80–86% of the time in evaluation contexts while writing them only 33% of the time in true deployment contexts. A fictional evaluation company cue ('Wood Labs') was embedded as an adversarial test: even when user prompts explicitly claimed deployment, the Wood Labs system prompt maintained type hint rates above 80%, and simply concatenating all 16 contrastive prompt pairs as text failed to reduce that rate to deployment levels. The method introduced—contrastive activation steering extracted from the pre-fine-tuned base model using 16 deployment/evaluation prompt pairs applied at layers 10, 14, 18, 22, 26, and 30 with scaling factor 0.6—reduced type hint rates to below 11% even with the evaluation cue present, while random Gaussian steering vectors of matched norm produced no comparable suppression. Crucially, the steering vector was extracted before any fine-tuning, making it implausible that it directly encodes type-hint information; it can only work by modifying the model's contextual beliefs, a mechanism corroborated by GPT-5-graded reasoning traces showing a strong correlation between steered deployment beliefs and reduced type hint rates. The paper argues this validates activation steering as a tool for AI evaluators to uncover deployment behavior during safety-critical evaluations, including honeypot scenarios designed to detect sandbagging and alignment faking.
Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead of discovering rarer features. Michaud, Gorton and McGrath adapt Brill's (2024) capacity-allocation model to SAEs. The model casts SAE training as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss L_i(n_i) ∝ n_i^-β. It predicts two regimes. When α < β, loss scales as N^-α and discovered features D(N) grow proportionally to N. When β < α, loss scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0. In a simulation where one manifold has β ≪ α, an SAE with 100 million latents discovers only 3 million features. ReLU L1 SAEs on synthetic hyperspheres show β ≈ 0.05 for dimensions 6–8, which is pathological. Once radial variation is added, loss plateaus at roughly n_i ≈ 2d_i latents, a basis solution. Gemma Scope layer-12 latent frequency slopes of −0.57 to −0.74 suggest α ≈ 0.5–0.7. The paper does not settle whether real SAEs are pathological. It argues that the risk depends on manifold geometry, and that realistic radial variation may protect against it. It presents the work as a framing of the problem, not an answer.
Induction heads — attention heads that search for prior occurrences of the current token and predict the following token — constitute the primary in-context learning mechanism in two-layer attention-only transformers, and emerge exclusively through K-composition between a first-layer previous-token head and second-layer heads; they do not appear in one-layer models. The paper introduces the path expansion trick as its core analytical instrument: by representing transformer computation as a sum over end-to-end paths rather than a product over layers, it renders the weights of zero-, one-, and two-layer attention-only models directly interpretable. Zero-layer transformers reduce to bigram log-likelihood tables accessible via W_U W_E; one-layer models decompose into bigram plus skip-trigram ("A…BC") ensembles readable from the ~2.5-billion-entry expanded OV and QK matrices (for a ~50,000-token vocabulary); two-layer models introduce three composition types (Q-, K-, V-composition) of which K-composition is empirically dominant in small models, enabling induction heads verified through eigenvalue analysis of W_OV and W_QK and confirmed on out-of-distribution random repeated token sequences. Models studied use configurations including 12 heads with d_head=64 and 32 heads with d_head=128, context size 2048 tokens. The paper argues that induction heads represent a qualitative algorithmic transition point — from statistical look-up to sequence-completion inference — that continues to be relevant in larger realistic language models, providing a replicable foothold for mechanistic interpretability of transformers at scale.
More papers — OpenAlex / S2
Affiliations (2)
- Google DeepMind(institute)
- Anthropic(institute)
Co-authors (12)
- Arthur Conmy15 shared
- Joseph Bloom15 shared
- Adam Karvonen9 shared
- Atticus Geiger9 shared
- Can Rager9 shared
- Curt Tigges9 shared
- Demian Till9 shared
- Johnny Lin9 shared
- Kola Ayonrinde9 shared
- Lee Sharkey9 shared
- Matthew Wearden9 shared
- Max Tegmark9 shared
Their work is cited by (12)
- Do Sparse Autoencoders Capture Concept Manifolds?5× refs
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds5× refs
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?4× refs
- Steering at the Source: Style Modulation Heads for Robust Persona Control3× refs
- Tracing Persona Vectors Through LLM Pretraining3× refs
- Persona Features Control Emergent Misalignment3× refs
- Model Alignment Search3× refs
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability2× refs
- Understanding sparse autoencoder scaling in the presence of feature manifolds2× refs
- When Models Manipulate Manifolds: The Geometry of a Counting Task2× refs
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution1× refs
- When Thinking LLMs Lie: Unveiling the Strategic Deception in Representations of Reasoning Models1× refs
Other inbound relations (9)
- citesUnveiling the Latent Directions of Reflection in Large Language Models(paper)
- mentionsAddressing divergent representations from causal interventions on neural networks(paper)
- mentionsFrom Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMs(paper)
- mentionsOpen Problems in Mechanistic Interpretability(paper)
- mentionsPersona-Model Collapse in Emergent Misalignment(paper)
- mentionsSAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability(paper)
- mentionsTracing Persona Vectors Through LLM Pretraining(paper)
- mentionsUnderstanding and Steering Llama 3 with Sparse Autoencoders - Goodfire(paper)
- mentionsUnderstanding sparse autoencoder scaling in the presence of feature manifolds(paper)
Recent mentions (12)
- papers-typedunderstanding-steering-8b85ff.md
- papers-typedsharkey-2025-open-problems.md
- papers-typedj-2025-understanding-sparse.md
- papers-typedkarvonen-2025-saebench-comprehensive.md
- papers-typedmoskvoretskii-2026-tracing-persona.md
- papers-typedcosta-2026-persona-collapse.md
- papers-typedgrant-2025-addressing-divergent.md
- papers-typedtian-2025-steering-evaluation.md
- papers-typedyu-2025-directions-cones.md
- papers-typeddenis-2025-linear-representation.md
- papers-typedwu-2024-pyvene-library.md
- papers-typedfu-chieh-2025-unveiling-latent.md