thinker:tom-mcgrathTom McGrath
Authored papers (2)
Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.
Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead of discovering rarer features. Michaud, Gorton and McGrath adapt Brill's (2024) capacity-allocation model to SAEs. The model casts SAE training as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss L_i(n_i) ∝ n_i^-β. It predicts two regimes. When α < β, loss scales as N^-α and discovered features D(N) grow proportionally to N. When β < α, loss scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0. In a simulation where one manifold has β ≪ α, an SAE with 100 million latents discovers only 3 million features. ReLU L1 SAEs on synthetic hyperspheres show β ≈ 0.05 for dimensions 6–8, which is pathological. Once radial variation is added, loss plateaus at roughly n_i ≈ 2d_i latents, a basis solution. Gemma Scope layer-12 latent frequency slopes of −0.57 to −0.74 suggest α ≈ 0.5–0.7. The paper does not settle whether real SAEs are pathological. It argues that the risk depends on manifold geometry, and that realistic radial variation may protect against it. It presents the work as a framing of the problem, not an answer.
More papers — OpenAlex / S2
Co-authors (12)
- Neel Nanda7 shared
- Atticus Geiger6 shared
- Lee Sharkey6 shared
- Max Tegmark6 shared
- Adrià Garriga-Alonso4 shared
- Arthur Conmy4 shared
- Bilal I. Chughtai4 shared
- Casper, Stephen4 shared
- Daniel Murfet4 shared
- David Anthony Bau4 shared
- Eric J. Michaud4 shared
- Eric Todd4 shared
Their work is cited by (8)
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds3× refs
- Do Sparse Autoencoders Capture Concept Manifolds?3× refs
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?2× refs
- Steering at the Source: Style Modulation Heads for Robust Persona Control2× refs
- Tracing Persona Vectors Through LLM Pretraining2× refs
- Model Alignment Search2× refs
- When Models Manipulate Manifolds: The Geometry of a Counting Task1× refs
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention1× refs
Other inbound relations (3)
- citesAre there any models of superposition which have a closed-form solution?(question)
- mentionsToy Models of Superposition(paper)
- mentionsUnderstanding sparse autoencoder scaling in the presence of feature manifolds(paper)
Recent mentions (4)
- papers-typedj-2025-understanding-sparse.md
- papers-typedsuperposition-c8171a.md
- papers-typedmckenzie-2026-endogenous-resistance.md
- papers-typedolah-2020-introduction.md