paper:arxiv-2209-10652Toy Models of Superposition
TL;DR
Neural networks store more sparse features than they have dimensions, and a one-line change shows it: adding a ReLU to the output of a linear W^T W autoencoder moves the optimum from PCA-like top-m feature selection to superposition. In toy models (n=20 features in m=5 dimensions with I_i=0.7^i; n=400 in m=30 for uniform runs), the ReLU output model matches the linear model on dense inputs but packs in extra features as sparsity rises. Features then settle into Thomson-problem polytopes (digons, triangles, tetrahedra, pentagons, square antiprisms). Dimensions-per-feature D* sticks at 1 and 1/2, with per-feature clusters at 3/4, 2/3, 2/5 and 3/8. Whether a feature is dropped, superposed or given its own dimension is set by a first-order phase change. The paper's instruments are the feature dimensionality D_i = ||W_i||²/Σ_j(Ŵ_i·W_j)², phase diagrams over sparsity and importance, and the W^TW polytope correspondence. A ReLU hidden-layer model computing abs(x) with n=100 features and m=40 neurons shows computation in superposition, via an asymmetric-superposition-plus-inhibition motif. Adversarial vulnerability rises more than 3x as superposition forms. The same account separates monosemantic from polysemantic neurons by sparsity and importance. Superposition is therefore a regime, not an inevitable cost, and may be removable at a performance price. Interpretability then reduces to enumerating features, by training superposition-free models (L1, MoE) or by recovering an overcomplete basis with sparse coding.
What to take away
- 1. A ReLU output model (x' = ReLU(W^TWx + b)) with n=20 features, m=5 dimensions and I_i=0.7^i matches the linear PCA-like solution on dense data but represents extra features non-orthogonally as sparsity increases.
- 2. Dimensions-per-feature D* = m/||W||_F² on the n=400, m=30 uniform model is sticky at 1 and 1/2, with the 1/2 plateau corresponding to antipodal pairs, a pattern the authors liken to the fractional quantum Hall effect.
- 3. Per-feature dimensionality D_i = ||W_i||²/Σ_j(Ŵ_i·W_j)² clusters at 3/4 (tetrahedron), 2/3 (triangle), 1/2 (antipodal pair), 2/5 (pentagon) and 3/8 (square antiprism), matching Thomson-problem solutions and tegum products of smaller polytopes.
- 4. In a 2-feature, 1-dimension toy-of-the-toy model, with the extra feature's importance varied from 0.1 to 10 and sparsity from 1.0 to 0.01 (ten runs per point, worst discarded), the choice among ignoring a feature, dedicating the dimension to a feature, or antipodal superposition [1,-1] switches discontinuously, and the loss curves cross, which confirms a first-order phase change.
- 5. A ReLU hidden-layer model with n=3 features and m=6 neurons computes abs(x)=ReLU(x)+ReLU(-x) with two neurons per feature, while at n=100, m=40 and I_i=0.8^i it computes abs in superposition once features are sparse enough, which shows computation, not just storage, can occur in superposition.
- 6. The asymmetric superposition motif stores two features with unequal weights (e.g. [2,-1/2]) and reciprocal output weights, and a second neuron inhibits the case of positive interference, converting damaging positive interference into harmless negative interference under the output ReLU.
- 7. L2 adversarial vulnerability (attack norm capped at 0.1 of average input norm, analytic optimal attack because gradient descent was masked at roughly 99% ReLU zeros) rises more than 3x as superposition forms and tracks features per dimension, while adversarial training needed attacks of 80% input L2 norm to eliminate superposition.
- 8. In correlated feature sets, features prefer orthogonal placement in separate tegum factors, forming local almost-orthogonal bases, and with six features in three correlated pairs, decreasing sparsity collapses each pair into its principal component until the solution equals PCA.
- 9. Methodology another lab could replicate: for the ReLU hidden-layer model with n=10, m=5 and I_i=0.75^i, train 1000 models per sparsity level and keep the lowest-loss solution, because small superposition models are hard to optimize and typical runs obscure the underlying structure.
- 10. The authors raise as an open question whether superposition survives scaling, i.e. whether it stays a constant fraction of features or grows, and they conjecture that MoE models, which turn neuron sparsity into saved flops, should show less superposition.
Peer brief — for seminar discussion
Toy Models of Superposition (Anthropic, transformer-circuits.pub 2022) asks when a network represents more features than it has dimensions. The setup is a synthetic autoencoder: sparse features x_i, each zero with probability S and otherwise uniform on [0,1], with importance I_i. They are projected by h=Wx and reconstructed as ReLU(W^Th+b) under importance-weighted MSE. The linear baseline learns the top-m principal components at any sparsity. The ReLU variant, with n=20, m=5 and I_i=0.7^i, starts representing extra features non-orthogonally as sparsity rises. This is the load-bearing finding, and it is obtained from a trained model whose ground-truth features are known. The method introduced is a set of instruments rather than a single algorithm: per-feature dimensionality D_i, phase diagrams over sparsity and importance, and the correspondence between polytopes and W^TW. With n=400 and m=30, D* sticks at 1 and 1/2, and features cluster at 3/4, 2/3, 2/5 and 3/8, which are Thomson-problem geometries. A ReLU-hidden abs(x) model (n=100, m=40) shows computation in superposition, and adversarial vulnerability rises more than 3x as superposition forms. The implication is that polysemanticity is the expected outcome for sparse features, so interpretability amounts to enumerating features. The three routes proposed are superposition-free models via L1 or mixture-of-experts, an overcomplete basis found by sparse coding, and hybrids. The paper's hypotheses are that superposition and polysemanticity increase with sparsity, that MoE should reduce superposition, and that it is linked to adversarial examples and grokking. A critical reader would push hardest on scope. The data are uniformly distributed, independent or hand-built correlated features, with importance given by hand and a clean ground truth that real language models do not supply. The authors concede that the geometry and learning-dynamics results are the most uncertain to transfer. A second concern is that the "phase change" is shown in tiny systems (2 features in 1 dimension) and uses a generalized notion of the term, so the 'sharpness' may not survive in large models. The polytope claims also depend on selecting the lowest-loss run from many seeds, because optimization is hard. An alternative to the hand-built toy autoencoder would be to analyze trained transformers directly, for example with dictionary learning on real activations, which is the route that the cited Sharkey et al. follow-up began on toy models. Validation against real models is limited to qualitative resemblance, such as InceptionV1 neurons becoming more polysemantic with depth and early transformer MLP neurons being highly polysemantic, which is suggestive but not a test.
Methods (7)
- Adversarial training (analytic attack)Reduces superposition but needs 80% L2 norm.
- Analytic optimal L2 attackUsed instead of gradient descent attacks due to gradient masking.
- Dimensions-per-feature metric D*=m/||W||_F^2Measures features learned.
- L1 regularization of hidden activationsRemoves superposition at cost of loss.
- Neuron weight stack plotVisualizes polysemantic neurons.
- Phase diagram experimentSweeps importance and sparsity to locate phase changes.
- W^TW visualizationVisualizes feature interference matrix.
Frameworks (9)
- Absolute value computation modelModel computing abs(x) in superposition.
- Compressed SensingMathematical framework enabling recovery of high-dimensional sparse vectors from low-dimensional projections; theoretically underpins sparse autoencoder approach
- DisentanglementRelated research agenda seeking representations that separate conceptually distinct factors; contrasted with superposition approach
- Hierarchy of feature propertiesDecomposability, linearity, superposition, basis alignment.
- Linear Representation HypothesisThe hypothesis that models internalize concepts as approximately linear directions in representation space; used to interpret MDS injection behavior
- ReLU hidden layer toy modelToy model with privileged basis.
- ReLU output toy modelx'=ReLU(W^TWx+b), main model demonstrating superposition.
- Sparse AutoencoderInterpretability framework used to decompose layer-40 activations into sparse feature sets for studying emotional alignment and persistence
- Superposition HypothesisCore theoretical framework: neural networks represent more features than neurons by encoding features as directions in superposition
Findings (22)
- Learning of 6 features in 3 dimensions with correlated sets proceeds through distinct geometric transformations ending in octahedron or antiprism
Learning dynamics.
- First-order phase change between not-learned, superposition, and dedicated dimension in 2-feature 1-dimension and 3-feature 2-dimension models
Both empirical and theoretical phase diagrams.
- Neurons in abs model have a primary feature with large weight plus secondary features with small weights, resembling language model neurons
Mirrors real networks.
- With n=5 features in m=2 dimensions, making one feature sufficiently sparse collapses pentagon into digons via first-order phase change
Perturbing a single feature.
- Feature dimensionalities cluster at 3/4 tetrahedron, 2/3 triangle, 1/2 antipodal, 2/5 pentagon, 3/8 square antiprism, and 0
Geometric configurations in uniform superposition.
- Correlated features prefer orthogonal placement in separate tegum factors; anticorrelated prefer negative interference
m=2 experiments.
- Superposition occurs in ReLU output toy models: more features than dimensions represented when features are sparse
Core demonstration.
- Correlated feature pairs collapse into principal components as sparsity decreases (6 features, hexagon to PCA)
Trade-off PCA vs superposition.
- Superposition begins with less important features in antipodal pairs, then transitions to other geometries
Toy model sparsity sweep.
- ReLU hidden layer model (n=10, m=5) shifts from monosemantic to polysemantic neurons as sparsity increases
Neuron-level phase change.
Claims (21)
- Non-uniform superposition can be understood as deformation of and jumping between uniform superposition configurations
Promising for real models.
- Superposition is a key challenge to using interpretability for AI safety because it blocks enumeration of features
Strategic picture.
- Privileged basis and superposition are two countervailing forces determining whether features align with neurons
Explanation for neuron interpretability.
- Superposition is not one thing: features with fractional dimensionality represent many distinct phases
Analogy to many ice phases.
- MoE models should have less superposition because they eat the gap between neuron and feature sparsity
Strategic Approach 1.
- Toy model predictions are consistent with empirical observations on polysemanticity in real networks
Validation; geometry and dynamics less certain.
- Because superposition is a phase, there may exist a regime for every model with no superposition
Cause for hope.
- Local bases are not enough for safety claims requiring full-distribution mechanistic accounts
Limit of local orthogonal bases.
- The phase diagram explains why neurons are sometimes monosemantic and sometimes polysemantic
Theory contribution.
- All neural codes, not just communication ones, could plausibly be compressed to some degree
Neuroscience implication.
Hypotheses (5)
- Adversarially robust models would have worse performance and be more interpretable because robustness requires giving up superposition
Predictions of superposition maximalism.
- Superposition hypothesis: networks represent more features than neurons as almost-orthogonal directions
Central hypothesis.
- Adversarial example transfer may be predicted by correlated arrangement of features in superposition
Speculative.
- Superposition and polysemanticity increase as feature sparsity increases
Major prediction checked against InceptionV1.
- Useful non-robust features could be an artifact of superposition
Open question linking to Goh.
Questions (13)
- Does the apparent phase change we observe in features/neurons have any connection to phase changes in compressed sensing?
Open question.
- Can we estimate the feature importance curve or feature sparsity curve of real models?
Open question.
- To what extent can neural networks do useful computation on features in superposition?
Open question.
- Why is it that neurons sometimes align with features and sometimes don't?
Motivating question.
- Are there any models of superposition which have a closed-form solution?
Open question.
- How can we control whether superposition and polysemanticity occur?
Most pragmatically important question.
- Should we expect superposition to go away if we just scale enough?
Open question.
- How many features should we expect to be stored in superposition?
Open question.
- How does superposition change if features are not independent?
Open question.
- Is there a statistical test for catching superposition?
Open question.
Original abstract (expand)
Demonstrates that hallucination in neural networks is a structural consequence of ungoverned superposition. Introduces a governance filter operating on the Gram matrix of the weight matrix that eliminates 100% of false-positive activations across all tested sparsity regimes (a toy-model feasibility result; true-positive retention not yet reported) in the Elhage et al. (2022) toy model framework, at a cost of 4-31% increased reconstruction error. Proposes a hierarchical two-supervisor architecture (routing supervisor + governance supervisor). No retraining, no extra parameters, applied as a post-hoc architectural layer. Originally a Google contest submission. --- **Author:** James E. Dunn — Independent Researcher, Hydrogen Lifecycle Research Programme **ORCID:** https://orcid.org/0009-0005-2679-6574 **Corpus (author search):** https://zenodo.org/search?q=creators.orcid:0009-0005-2679-6574 **SciX (NASA discovery):** https://scixplorer.org/search/q=orcid%3A0009-0005-2679-6574 **ADS (Harvard-CfA):** https://ui.adsabs.harvard.edu/search/q=orcid%3A0009-0005-2679-6574 **License:** CC BY 4.0 International
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Superposition as Lossy Compression: Measure with Sparse Autoencoders and Connect to Adversarial VulnerabilityLeonard Bereska and Zoe Tzifa-Kratira and Reza Samavi and Efstratios Gavves2025≈ 89%
- From Data Statistics to Feature Geometry: How Correlations Shape SuperpositionEdward Stevinson, Melih Barsbey, Tolga Birdal, Pedro A.M. Mediano Lucas Prieto2026≈ 88%
- ≈ 87%
- ≈ 84%
- ≈ 84%
- Fundamental Limits of Neural Network Sparsification: Evidence from Catastrophic Interpretability CollapseRajiv Misra, Sanjay Kumar Singh Dip Roy2026≈ 83%
- Disentangling Polysemantic Neurons with a Null-Calibrated Polysemanticity Index and Causal Patch InterventionsDhruv Kumar Manan Gupta2025≈ 83%
- Superscopes: Amplifying Internal Feature Representations for Language Model InterpretationJonathan Jacobi and Gal Niv2025≈ 83%
- Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision TransformersSiyu Zhang2026≈ 83%
- A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious MinimaHarshvardhan Saini, Zhaoqian Yao, Zheng Lin, Yizhen Liao, Jingyi Cui, Yisen Wang, Mengnan Du, Dianbo Liu Yiming Tang2026≈ 83%
- Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series ForecastingAlper Y{\i}ld{\i}r{\i}m2026≈ 82%
- The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language ModelsGuillaume Rabusseau, Marius Mosbach Michael Rizvi-Martel2026≈ 82%
- SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMsAndrew Gordon, Peter Lai, Xavier Poncini, David Quarel, and Stefan Heimersheim Sean P. Fillingham2025≈ 82%
- Constructing Interpretable Features from Compositional Neuron GroupsAtticus Geiger, Mor Geva Or Shafran2026≈ 82%
- ≈ 82%
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentionin corpus2026≈ 81%
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?in corpus2025≈ 81%
- ≈ 81%
- PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant CircuitsErblina Purelku, Johanna Vielhaben, Wojciech Samek, Sebastian Lapuschkin Maximilian Dreyer2024≈ 81%
- Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future DirectionsUsman Naseem2026≈ 81%
- Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical AblationGabriele Dominici, Marc Langheinrich Francesco Sovrano2026≈ 81%
- Insights on Crosscoder Model Diffingin corpus≈ 81%
- The Platonic Representation Hypothesisin corpus2024≈ 81%
- Stochastic Parameter Decompositionin corpus2025≈ 81%
- Model Alignment Searchin corpus2025≈ 80%
- A Mathematical Framework for Transformer Circuitsin corpus2021≈ 80%
- ≈ 80%
- Transformer Circuits Threadcited2022
+2 more
Similar preprints — Semantic Scholar
Cited by (11)
- Understanding and Steering Llama 3 with Sparse Autoencoders - Goodfire
A sparse autoencoder trained on layer-19 residual-stream activations of Llama-3-8B-Instruct, using LMSYS-Chat-1M, yields features that can be labeled automatically, ranked by causal relevance, and use
- Adversarial Examples Are Not Bugs, They Are Superposition
Superposition causally contributes to adversarial vulnerability, and the link runs in both directions: in a toy ReLU autoencoder (n=100 features, m=20 hidden units), raising feature sparsity raises fe
- Understanding sparse autoencoder scaling in the presence of feature manifolds
Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead
- Do Sparse Autoencoders Capture Concept Manifolds?
Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept
- Addressing divergent representations from causal interventions on neural networks
Causal intervention methods central to mechanistic interpretability—including activation patching, mean-difference vector patching, Sparse Autoencoders, and Distributed Alignment Search (DAS)—systemat
- From Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMs
Propositional truth in LLMs is not encoded as a single linear direction but as a multi-dimensional subspace that can be characterized by concept cones—sets of all nonnegative linear combinations of or
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
At sufficient scale, LLMs linearly represent the truth or falsehood of factual statements in their internal activations — a claim supported by PCA visualizations, cross-dataset probe transfer, and cau
- Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders
Applying TopK Sparse Autoencoders (SAEs) to three architecturally distinct EEG foundation models — SleepFM, REVE, and LaBraM — reveals that clinical concepts are not cleanly separable in these models'
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Manifold steering — intervening on model activations along paths constrained to lie on a learned activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectorie
- The Guanyin Protocol: A Framework for Immediately Establishing an Understanding of Both Causality and Compassion in LLM Systems Using Semantic Anchoring
Semantic anchoring — the binding of a pretrained model's latent patterns to task-specific targets via external structure — predicts threshold-like performance flips with a single calibrated score S =
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Training
Probe-based data attribution, introduced here as a method for surfacing and mitigating undesirable post-training behaviors, reduces harmful compliance in OLMo 2 7B by 63% through datapoint filtering a