thinker:roger-grosseRoger Grosse
Unpublished independent model diffing work.
Authored papers (2)
Neural networks store more sparse features than they have dimensions, and a one-line change shows it: adding a ReLU to the output of a linear W^T W autoencoder moves the optimum from PCA-like top-m feature selection to superposition. In toy models (n=20 features in m=5 dimensions with I_i=0.7^i; n=400 in m=30 for uniform runs), the ReLU output model matches the linear model on dense inputs but packs in extra features as sparsity rises. Features then settle into Thomson-problem polytopes (digons, triangles, tetrahedra, pentagons, square antiprisms). Dimensions-per-feature D* sticks at 1 and 1/2, with per-feature clusters at 3/4, 2/3, 2/5 and 3/8. Whether a feature is dropped, superposed or given its own dimension is set by a first-order phase change. The paper's instruments are the feature dimensionality D_i = ||W_i||²/Σ_j(Ŵ_i·W_j)², phase diagrams over sparsity and importance, and the W^TW polytope correspondence. A ReLU hidden-layer model computing abs(x) with n=100 features and m=40 neurons shows computation in superposition, via an asymmetric-superposition-plus-inhibition motif. Adversarial vulnerability rises more than 3x as superposition forms. The same account separates monosemantic from polysemantic neurons by sparsity and importance. Superposition is therefore a regime, not an inevitable cost, and may be removable at a performance price. Interpretability then reduces to enumerating features, by training superposition-free models (L1, MoE) or by recovering an overcomplete basis with sparse coding.
More papers — OpenAlex / S2
Originates (1)
Studies (1)
Co-authors (12)
- Bushnaq, Lucius2 shared
- Lewis, Owen2 shared
- Merullo, Jack2 shared
- Vatsavaya, Srihita2 shared
- Carol Chen1 shared
- Catherine Olsson1 shared
- Chris Olah1 shared
- Christopher Olah1 shared
- Dario Amodei1 shared
- David Yunis1 shared
- Dawn Drain1 shared
- Elhage, Nelson1 shared
Their work is cited by (11)
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets1× refs
- Addressing divergent representations from causal interventions on neural networks1× refs
- From Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMs1× refs
- The Guanyin Protocol: A Framework for Immediately Establishing an Understanding of Both Causality and Compassion in LLM Systems Using Semantic Anchoring1× refs
- Adversarial Examples Are Not Bugs, They Are Superposition1× refs
- Understanding sparse autoencoder scaling in the presence of feature manifolds1× refs
- Do Sparse Autoencoders Capture Concept Manifolds?1× refs
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior1× refs
- Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders1× refs
- Understanding and Steering Llama 3 with Sparse Autoencoders - Goodfire1× refs
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Training1× refs
Other inbound relations (1)
Recent mentions (2)
- papers-typedjack-2025-memorization-reasoning.md
- papers-typed
authors-sparse-crosscoders.md