thinker:arthur-conmyArthur Conmy
Authored papers (2)
Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.
Gains on the sparsity-fidelity frontier do not predict how well a sparse autoencoder performs on downstream tasks, and SAEBench demonstrates this by scoring over 200 SAEs on eight metrics. Matryoshka BatchTopK SAEs rank below TopK and BatchTopK on loss recovered, yet they take the best score on Absorption, RAVEL, Sparse Probing, SCR and TPP, which is 5 of 8 metrics, in the typical L0 range of 40-200. Matryoshka is also the only architecture whose disentanglement scores improve when width grows from 4k to 16k to 65k latents on Gemma-2-2B (layer 12). The other architectures show inverse scaling on absorption and SCR, and the effect persists when the number of ablated latents is held fixed. SAEBench combines four capability groups (concept detection, interpretability, reconstruction, feature disentanglement). It adds two novel zero-ablation metrics, Spurious Correlation Removal and Targeted Probe Perturbation, and it extends the absorption metric to partial and shared absorption. The suite covers seven architectures (ReLU, TopK, BatchTopK, Gated, JumpReLU, P-annealing, Matryoshka), Gemma-2-2B and Pythia-160M, and Gemma-Scope SAEs up to 1M latents. Hierarchical architectures beat the rest by 30-40% on SCR and absorption. No single L0 wins everywhere, though L0 of 50-150 is a reasonable compromise. The implication is that SAE progress needs a multi-metric benchmark, and that optimizing reconstruction at fixed sparsity can hide failures in concept isolation, which grow with scale. The paper declines to combine the metrics into one score.
More papers — OpenAlex / S2
Co-authors (12)
- Neel Nanda15 shared
- Joseph Bloom13 shared
- Adam Karvonen9 shared
- Can Rager9 shared
- Curt Tigges9 shared
- Demian Till9 shared
- Johnny Lin9 shared
- Kola Ayonrinde9 shared
- Matthew Wearden9 shared
- Atticus Geiger6 shared
- Callum Stuart McDougall6 shared
- Chanin, David6 shared
Their work is cited by (6)
- Model Alignment Search2× refs
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?2× refs
- Steering at the Source: Style Modulation Heads for Robust Persona Control2× refs
- Do Sparse Autoencoders Capture Concept Manifolds?2× refs
- Tracing Persona Vectors Through LLM Pretraining2× refs
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds2× refs
Other inbound relations (1)
Recent mentions (1)
- papers-typedkarvonen-2025-saebench-comprehensive.md