thinker
pending-review
thinker:orcid-0000-0003-0904-4862

MARTIN P. WATTENBERG

Authored
2
Introduces
0
Studies
0
Affiliations
0
Cited by
12

Authored papers (2)

  • Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.

  • Neural networks store more sparse features than they have dimensions, and a one-line change shows it: adding a ReLU to the output of a linear W^T W autoencoder moves the optimum from PCA-like top-m feature selection to superposition. In toy models (n=20 features in m=5 dimensions with I_i=0.7^i; n=400 in m=30 for uniform runs), the ReLU output model matches the linear model on dense inputs but packs in extra features as sparsity rises. Features then settle into Thomson-problem polytopes (digons, triangles, tetrahedra, pentagons, square antiprisms). Dimensions-per-feature D* sticks at 1 and 1/2, with per-feature clusters at 3/4, 2/3, 2/5 and 3/8. Whether a feature is dropped, superposed or given its own dimension is set by a first-order phase change. The paper's instruments are the feature dimensionality D_i = ||W_i||²/Σ_j(Ŵ_i·W_j)², phase diagrams over sparsity and importance, and the W^TW polytope correspondence. A ReLU hidden-layer model computing abs(x) with n=100 features and m=40 neurons shows computation in superposition, via an asymmetric-superposition-plus-inhibition motif. Adversarial vulnerability rises more than 3x as superposition forms. The same account separates monosemantic from polysemantic neurons by sparsity and importance. Superposition is therefore a regime, not an inevitable cost, and may be removable at a performance price. Interpretability then reduces to enumerating features, by training superposition-free models (L1, MoE) or by recovering an overcomplete basis with sparse coding.

More papers — OpenAlex / S2

Co-authors (12)