paper:doi-10-48550-arxiv-2508-17456Adversarial Examples Are Not Bugs, They Are Superposition
TL;DR
Superposition causally contributes to adversarial vulnerability, and the link runs in both directions: in a toy ReLU autoencoder (n=100 features, m=20 hidden units), raising feature sparsity raises features-per-dimension and adversarial vulnerability (r≈0.99 under Elhage-style attacks), and adversarial training with α=0.5, ε=0.1‖x‖₂ for 150,000 steps lowers features-per-dimension at matched sparsity. The same direction holds in real networks: robust ImageNet ResNet18s from Salman et al. (2020) trained at L2 ε∈{0,1,3,5} give lower SAE reconstruction MSE at fixed TopK sparsity (k=8,16,32), read as a proxy for less superposition. Gorton and Lewis supply the evidence by pairing a toy-model setup in which superposition is exactly measurable and controllable via sparsity with SAE loss as a proxy in ResNet18. Geometry supports the mechanism: non-robust toy models show about 2× the mean off-diagonal interference (WᵀW) of robust ones, and PGD attacks raise SAE L0 at layer4 from 126 to 351 (2.8×), whereas FGSM barely moves it (1.02×). Robust models also look more robust than their superposition level predicts, which suggests interference structure matters beyond its total amount. The causal claim in real models is only one-directional, because superposition cannot be intervened on there. The paper argues superposition is at least one causal factor and possibly the primary one. If so, robustness trades off against the capacity superposition provides, and interpretability and robustness research share objectives.
What to take away
- 1. In toy models with n=100 features compressed into m=20 hidden units, vulnerability to a noise-plus-one-step-gradient L2 attack (ε=10% of mean input norm) rises with feature sparsity and with features per dimension.
- 2. Under Elhage-style analytic attacks, features per dimension correlates with adversarial vulnerability at r≈0.99 (p<0.0001) across 30 toy models trained at exponentially spaced densities from 1.0 to 0.1.
- 3. Adversarially training toy models (α=0.5 clean/adversarial mix, ε=0.1‖x‖₂, 150,000 steps, lr 1e-3) reduces features per dimension at a given sparsity, which the authors describe as the first demonstration of causality from robustness to superposition.
- 4. Non-robust toy models have about 2× the mean off-diagonal interference in WᵀW of robust models trained at the same sparsity, and adversarial inputs light up many interfering features in the superposition graph while clean inputs light up few.
- 5. On ImageNet ResNet18s from Salman et al. (2020) with L2 training ε∈{0,1,3,5}, TopK SAEs with k∈{8,16,32} reach lower reconstruction MSE on more robust models at fixed sparsity, which the paper reads as evidence that robustness reduces superposition.
- 6. PGD adversarial examples raise SAE L0 at ResNet18 layer4 from 126.0 to 351.4 (2.8×) on n=100,000 images, whereas FGSM barely changes it (128.1, about 1.02×), consistent with iterative attacks accumulating interference.
- 7. Methodology another lab could replicate: use feature sparsity as an instrumental variable to set superposition, then confirm the manipulation works by checking features per dimension before reading off robustness, and cross-check with adversarially trained models, which shift superposition independently of the data distribution.
- 8. The toy-model vulnerability curve has an unexplained dip at antipodal superposition (phase transitions near 1.7 and 4 on the 1/(1−S) axis), where vulnerability drops temporarily.
- 9. Open question: robust toy models are more robust than their total superposition level predicts, and the authors conjecture that the structure of the interference matrix, not only its overall level, determines vulnerability.
- 10. If superposition is a primary cause, then robustness necessarily costs the capacity superposition provides, so the paper predicts an unavoidable trade-off and an alignment between interpretability and robustness goals.
Peer brief — for seminar discussion
This work tests Elhage et al.'s 2022 conjecture that superposition drives adversarial examples. The method is a bidirectional intervention design. In a toy ReLU autoencoder with 100 features and 20 hidden units, superposition is exactly measurable as features per dimension and controllable through feature sparsity. In ResNet18 it is not measurable, so the work substitutes sparse-autoencoder reconstruction loss as a proxy. The headline results are four. Vulnerability to a noise-plus-one-step L2 gradient attack rises with sparsity and superposition; Elhage-style analytic attacks give r≈0.99 across 30 models. Adversarial training (α=0.5, 150k steps) lowers features per dimension at matched sparsity. Non-robust toy models carry about 2× the off-diagonal interference of robust ones. Robust ImageNet ResNet18s from Salman et al. (L2 ε up to 5) get lower TopK SAE MSE at k=8, 16 and 32. A further analysis shows PGD raising SAE L0 at layer4 from 126 to 351 (2.8×), while FGSM changes it by about 1.02×. The load-bearing claim is that superposition is at least one causal factor. In real networks the causal arrow is shown only from robustness to the proxy, never from superposition to robustness. The implication is a capacity-robustness trade-off, with interpretability and robustness research pulling in the same direction. A natural alternative to the SAE-loss proxy would be measuring feature interference directly, for instance through decoder-direction coherence, or using circuit-level ablations. The unexplained antipodal dip and the finding that robust models beat their superposition level together suggest interference structure matters more than a scalar count. The strongest pushback is on the real-model evidence. SAE reconstruction loss depends on more than superposition. Adversarial training changes activation statistics, smoothness and effective dimensionality, and plausibly changes feature density and importance, any of which could lower MSE without reducing superposition. Standardizing activations per model does not resolve that. The text says it has no hypothesis for a competing explanation, which is an absence of imagination rather than evidence. The toy-model analysis also uses uniform feature importance, and the toy attack is one-step. The paper's argument that data-distribution confounds bias against its trend is asserted rather than shown. The paper's hypothesis is that superposition is a major, possibly primary, cause. The data fit that claim but cannot rank it against other factors.
Methods (11)
- Elhage analytic optimal feature attackAnalytically constructed per-feature attack.
- Features per dimensionMeasure of superposition in toy models.
- FGSMSingle-step attack.
- L0 active-feature countMeasure of features activated by clean vs adversarial inputs.
- L1 ReLU SAESAE variant trained on ResNet18.
- Mixed clean/adversarial training protocol (α=0.5)Trains toy models for 150,000 steps with L2 ε=0.1‖x‖.
- Noise-plus-one-step gradient L2 attackPrimary attack used for toy model robustness.
- PGDMulti-step attack.
- SAE reconstruction loss as superposition proxyProxy since superposition cannot be directly measured in real models.
- sparse autoencodersExisting method for model interpretability that decodes model activations rather than parameters themselves, noted as incomplete solution.
- Superposition geometry graphGraph with edges (Wi·Wj)^2 to visualize interference.
Frameworks (5)
- Compressed SensingMathematical framework enabling recovery of high-dimensional sparse vectors from low-dimensional projections; theoretically underpins sparse autoencoder approach
- Interference minimization view of trainingAverage-case vs worst-case interference objective conceptualization.
- Robust features hypothesisModels exploit non-robust but predictive patterns.
- Superposition HypothesisCore theoretical framework: neural networks represent more features than neurons by encoding features as directions in superposition
- TopK SAEEvaluated architecture (Gao et al.).
Datasets (2)
- ImageNetLarge-scale image database used for pre-training.
- Robust ImageNet ResNet18 models (Salman et al.)Pretrained models with varying attack sizes.
Findings (12)
- Robust and non-robust toy models trained at the same sparsity have strikingly similar superposition geometries, with robust ones having less superposition.
Figure 4.
- r≈0.99 (p<0.0001) correlation between features per dimension and adversarial vulnerability in toy models with Elhage-style attacks.
Appendix replication.
- In toy models, adversarial vulnerability increases with feature sparsity and superposition, with a dip at antipodal superposition.
Superposition controls robustness.
- Adversarial training reduces features per dimension and adversarial vulnerability in toy models at given sparsity levels.
Robustness controls superposition.
- More robust ResNet18 models consistently achieve lower TopK SAE reconstruction loss at a given sparsity level.
Evidence in real models.
- Adversarial examples activate more features than clean inputs across all ResNet18 layers.
Figure 6.
- Non-robust toy models have mean off-diagonal interference about 2× that of robust models.
Figure 3B.
- Robust toy models are often more robust than expected for their superposition level.
Unexpected result.
- FGSM L0 drops near baseline (1.02×) at layer4 (128.1 vs 126.0).
Contrast with PGD.
- FGSM increases L0 by about 1.3× at conv1 (46.2 vs 36.0).
Early layer effect.
Claims (10)
- If superposition is a primary cause, there is a fundamental unavoidable trade-off between robustness and model capacity, and interpretability and robustness are aligned.
Discussion implication.
- Results rely on proxies and could be consistent with multiple causal factors; only the potentiality of a causal relationship is established in real models.
Stated limitation.
- Adversarial attacks work by activating many interfering features in superposition; adversarial training reorganizes geometry to prevent this.
Interpretation of Figure 3.
- Adversarial training reduces superposition; these are the first results demonstrating causality from robustness to superposition.
Novelty claim.
- Overall superposition level doesn't tell the full story; some superposition structures are more vulnerable than others.
Explains robust models being more robust than expected.
- Changing feature density across models likely causes underestimation of the superposition-vulnerability relationship.
Confound addressed.
- Superposition is at least one causal factor in the existence of adversarial examples, and possibly the primary one.
Central thesis.
- Superposition is sufficient to theoretically explain all adversarial phenomena the authors are aware of.
Supported by Table 1.
- Iterative attacks like PGD more effectively exploit accumulated interference in deeper representations.
Interpretation of layer4 L0 spike.
- Lower SAE loss at fixed sparsity likely indicates less interference and less superposition.
Justifies SAE proxy.
Hypotheses (3)
- Superposition may be a major contributing factor, or even the primary cause, of adversarial examples.
Hypothesis under test.
- If adversarial attacks exploit interference, they should activate more features than clean inputs.
Prediction tested via L0.
- Robustness influences superposition, predicting lower SAE loss in more robust models.
Popperian credit for surprising prediction.
Questions (2)
Original abstract (expand)
Adversarial examples -- inputs with imperceptible perturbations that fool neural networks -- remain one of deep learning's most perplexing phenomena despite nearly a decade of research. While numerous defenses and explanations have been proposed, there is no consensus on the fundamental mechanism. One underexplored hypothesis is that superposition, a concept from mechanistic interpretability, may be a major contributing factor, or even the primary cause. We present four lines of evidence in support of this hypothesis, greatly extending prior arguments by Elhage et al. (2022): (1) superposition can theoretically explain a range of adversarial phenomena, (2) in toy models, intervening on superposition controls robustness, (3) in toy models, intervening on robustness (via adversarial training) controls superposition, and (4) in ResNet18, intervening on robustness (via adversarial training) controls superposition.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Toy Models of Superpositioncitedin corpus2022≈ 87%
- ≈ 81%
- Superposition as Lossy Compression: Measure with Sparse Autoencoders and Connect to Adversarial VulnerabilityLeonard Bereska and Zoe Tzifa-Kratira and Reza Samavi and Efstratios Gavves2025≈ 86%
- From Data Statistics to Feature Geometry: How Correlations Shape SuperpositionEdward Stevinson, Melih Barsbey, Tolga Birdal, Pedro A.M. Mediano Lucas Prieto2026≈ 84%
- Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned TransformersSanthosh Kumar Ravindran2025≈ 82%
- ≈ 82%
- ≈ 82%
- Evaluating Similitude and Robustness of Deep Image Denoising Models via Adversarial AttackJiebao Sun, Yao Li, Zhichang Guo, Wangmeng Zuo Jie Ning2026≈ 82%
- How Adversarial Environments Mislead Agentic AI?Huichi Zhou, Zhenhao Li, Peiyuan Jing, Krinos Li, Hamed Haddadi Zhonghao Zhan2026≈ 82%
- SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMsAndrew Gordon, Peter Lai, Xavier Poncini, David Quarel, and Stefan Heimersheim Sean P. Fillingham2025≈ 82%
- Improving Alignment and Robustness with Circuit BreakersLong Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, Dan Hendrycks Andy Zou2024≈ 81%
- Probing the Robustness of Large Language Models Safety to Latent PerturbationsKexin Huang, Zongqi Wang, Yixu Wang, Jie Li, Yuanqi Yao, Yang Yao, Yujiu Yang, Yan Teng, Yingchun Wang Tianle Gu2025≈ 81%
- Falsifying Sparse Autoencoder Reasoning Features in Language ModelsZhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi George Ma2026≈ 81%
- Improving Robustness In Sparse Autoencoders via Masked RegularizationKowshik Thopalli, Bhavya Kailkhura and Wesam Sakla Vivek Narayanaswamy2026≈ 81%
- ≈ 81%
- Adversarial Lens: Exploiting Attention Layers to Generate Adversarial Examples for EvaluationKaustubh Dhole2026≈ 81%
- ≈ 81%
- The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling LawsMaciej Trzaskowski, Quan Nguyen, Fred Roosta Eslam Zaher2026≈ 81%
- Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series ForecastingAlper Y{\i}ld{\i}r{\i}m2026≈ 80%
- ≈ 80%
- Unpacking Robustness in Inflectional Languages: Adversarial Evaluation and Mechanistic InsightsMarek Klonowski, Marcin Oleksy and Arkadiusz Janz Pawe{\l} Walkowiak2025≈ 80%
- ≈ 80%
- ≈ 80%
- Alignment faking in large language modelsin corpus2024≈ 80%
- Insights on Crosscoder Model Diffingin corpus≈ 80%
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?in corpus2025≈ 79%
- Persona Features Control Emergent Misalignmentin corpus2025≈ 79%
- ≈ 79%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 79%
- ≈ 67%
+26 more