paper
active
2022
ⓒ 49
paper:arxiv-2209-10652

Toy Models of Superposition

TL;DR

Neural networks store more sparse features than they have dimensions, and a one-line change shows it: adding a ReLU to the output of a linear W^T W autoencoder moves the optimum from PCA-like top-m feature selection to superposition. In toy models (n=20 features in m=5 dimensions with I_i=0.7^i; n=400 in m=30 for uniform runs), the ReLU output model matches the linear model on dense inputs but packs in extra features as sparsity rises. Features then settle into Thomson-problem polytopes (digons, triangles, tetrahedra, pentagons, square antiprisms). Dimensions-per-feature D* sticks at 1 and 1/2, with per-feature clusters at 3/4, 2/3, 2/5 and 3/8. Whether a feature is dropped, superposed or given its own dimension is set by a first-order phase change. The paper's instruments are the feature dimensionality D_i = ||W_i||²/Σ_j(Ŵ_i·W_j)², phase diagrams over sparsity and importance, and the W^TW polytope correspondence. A ReLU hidden-layer model computing abs(x) with n=100 features and m=40 neurons shows computation in superposition, via an asymmetric-superposition-plus-inhibition motif. Adversarial vulnerability rises more than 3x as superposition forms. The same account separates monosemantic from polysemantic neurons by sparsity and importance. Superposition is therefore a regime, not an inevitable cost, and may be removable at a performance price. Interpretability then reduces to enumerating features, by training superposition-free models (L1, MoE) or by recovering an overcomplete basis with sparse coding.

What to take away

  1. 1. A ReLU output model (x' = ReLU(W^TWx + b)) with n=20 features, m=5 dimensions and I_i=0.7^i matches the linear PCA-like solution on dense data but represents extra features non-orthogonally as sparsity increases.
  2. 2. Dimensions-per-feature D* = m/||W||_F² on the n=400, m=30 uniform model is sticky at 1 and 1/2, with the 1/2 plateau corresponding to antipodal pairs, a pattern the authors liken to the fractional quantum Hall effect.
  3. 3. Per-feature dimensionality D_i = ||W_i||²/Σ_j(Ŵ_i·W_j)² clusters at 3/4 (tetrahedron), 2/3 (triangle), 1/2 (antipodal pair), 2/5 (pentagon) and 3/8 (square antiprism), matching Thomson-problem solutions and tegum products of smaller polytopes.
  4. 4. In a 2-feature, 1-dimension toy-of-the-toy model, with the extra feature's importance varied from 0.1 to 10 and sparsity from 1.0 to 0.01 (ten runs per point, worst discarded), the choice among ignoring a feature, dedicating the dimension to a feature, or antipodal superposition [1,-1] switches discontinuously, and the loss curves cross, which confirms a first-order phase change.
  5. 5. A ReLU hidden-layer model with n=3 features and m=6 neurons computes abs(x)=ReLU(x)+ReLU(-x) with two neurons per feature, while at n=100, m=40 and I_i=0.8^i it computes abs in superposition once features are sparse enough, which shows computation, not just storage, can occur in superposition.
  6. 6. The asymmetric superposition motif stores two features with unequal weights (e.g. [2,-1/2]) and reciprocal output weights, and a second neuron inhibits the case of positive interference, converting damaging positive interference into harmless negative interference under the output ReLU.
  7. 7. L2 adversarial vulnerability (attack norm capped at 0.1 of average input norm, analytic optimal attack because gradient descent was masked at roughly 99% ReLU zeros) rises more than 3x as superposition forms and tracks features per dimension, while adversarial training needed attacks of 80% input L2 norm to eliminate superposition.
  8. 8. In correlated feature sets, features prefer orthogonal placement in separate tegum factors, forming local almost-orthogonal bases, and with six features in three correlated pairs, decreasing sparsity collapses each pair into its principal component until the solution equals PCA.
  9. 9. Methodology another lab could replicate: for the ReLU hidden-layer model with n=10, m=5 and I_i=0.75^i, train 1000 models per sparsity level and keep the lowest-loss solution, because small superposition models are hard to optimize and typical runs obscure the underlying structure.
  10. 10. The authors raise as an open question whether superposition survives scaling, i.e. whether it stays a constant fraction of features or grows, and they conjecture that MoE models, which turn neuron sparsity into saved flops, should show less superposition.

Peer brief — for seminar discussion

Toy Models of Superposition (Anthropic, transformer-circuits.pub 2022) asks when a network represents more features than it has dimensions. The setup is a synthetic autoencoder: sparse features x_i, each zero with probability S and otherwise uniform on [0,1], with importance I_i. They are projected by h=Wx and reconstructed as ReLU(W^Th+b) under importance-weighted MSE. The linear baseline learns the top-m principal components at any sparsity. The ReLU variant, with n=20, m=5 and I_i=0.7^i, starts representing extra features non-orthogonally as sparsity rises. This is the load-bearing finding, and it is obtained from a trained model whose ground-truth features are known. The method introduced is a set of instruments rather than a single algorithm: per-feature dimensionality D_i, phase diagrams over sparsity and importance, and the correspondence between polytopes and W^TW. With n=400 and m=30, D* sticks at 1 and 1/2, and features cluster at 3/4, 2/3, 2/5 and 3/8, which are Thomson-problem geometries. A ReLU-hidden abs(x) model (n=100, m=40) shows computation in superposition, and adversarial vulnerability rises more than 3x as superposition forms. The implication is that polysemanticity is the expected outcome for sparse features, so interpretability amounts to enumerating features. The three routes proposed are superposition-free models via L1 or mixture-of-experts, an overcomplete basis found by sparse coding, and hybrids. The paper's hypotheses are that superposition and polysemanticity increase with sparsity, that MoE should reduce superposition, and that it is linked to adversarial examples and grokking. A critical reader would push hardest on scope. The data are uniformly distributed, independent or hand-built correlated features, with importance given by hand and a clean ground truth that real language models do not supply. The authors concede that the geometry and learning-dynamics results are the most uncertain to transfer. A second concern is that the "phase change" is shown in tiny systems (2 features in 1 dimension) and uses a generalized notion of the term, so the 'sharpness' may not survive in large models. The polytope claims also depend on selecting the lowest-loss run from many seeds, because optimization is hard. An alternative to the hand-built toy autoencoder would be to analyze trained transformers directly, for example with dictionary learning on real activations, which is the route that the cited Sharkey et al. follow-up began on toy models. Validation against real models is limited to qualitative resemblance, such as InceptionV1 neurons becoming more polysemantic with depth and early transformer MLP neurons being highly polysemantic, which is suggestive but not a test.

Methods (7)

Frameworks (9)

  • Absolute value computation model
    Model computing abs(x) in superposition.
  • Compressed Sensing
    Mathematical framework enabling recovery of high-dimensional sparse vectors from low-dimensional projections; theoretically underpins sparse autoencoder approach
  • Disentanglement
    Related research agenda seeking representations that separate conceptually distinct factors; contrasted with superposition approach
  • Hierarchy of feature properties
    Decomposability, linearity, superposition, basis alignment.
  • Linear Representation Hypothesis
    The hypothesis that models internalize concepts as approximately linear directions in representation space; used to interpret MDS injection behavior
  • ReLU hidden layer toy model
    Toy model with privileged basis.
  • ReLU output toy model
    x'=ReLU(W^TWx+b), main model demonstrating superposition.
  • Sparse Autoencoder
    Interpretability framework used to decompose layer-40 activations into sparse feature sets for studying emotional alignment and persistence
  • Superposition Hypothesis
    Core theoretical framework: neural networks represent more features than neurons by encoding features as directions in superposition

Findings (22)

Original abstract (expand)

Demonstrates that hallucination in neural networks is a structural consequence of ungoverned superposition. Introduces a governance filter operating on the Gram matrix of the weight matrix that eliminates 100% of false-positive activations across all tested sparsity regimes (a toy-model feasibility result; true-positive retention not yet reported) in the Elhage et al. (2022) toy model framework, at a cost of 4-31% increased reconstruction error. Proposes a hierarchical two-supervisor architecture (routing supervisor + governance supervisor). No retraining, no extra parameters, applied as a post-hoc architectural layer. Originally a Google contest submission. --- **Author:** James E. Dunn — Independent Researcher, Hydrogen Lifecycle Research Programme **ORCID:** https://orcid.org/0009-0005-2679-6574 **Corpus (author search):** https://zenodo.org/search?q=creators.orcid:0009-0005-2679-6574 **SciX (NASA discovery):** https://scixplorer.org/search/q=orcid%3A0009-0005-2679-6574 **ADS (Harvard-CfA):** https://ui.adsabs.harvard.edu/search/q=orcid%3A0009-0005-2679-6574 **License:** CC BY 4.0 International

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+2 more

Similar preprints — Semantic Scholar

Cited by (11)