paper
active
2025
ⓒ 1
paper:doi-10-48550-arxiv-2508-17456

Adversarial Examples Are Not Bugs, They Are Superposition

TL;DR

Superposition causally contributes to adversarial vulnerability, and the link runs in both directions: in a toy ReLU autoencoder (n=100 features, m=20 hidden units), raising feature sparsity raises features-per-dimension and adversarial vulnerability (r≈0.99 under Elhage-style attacks), and adversarial training with α=0.5, ε=0.1‖x‖₂ for 150,000 steps lowers features-per-dimension at matched sparsity. The same direction holds in real networks: robust ImageNet ResNet18s from Salman et al. (2020) trained at L2 ε∈{0,1,3,5} give lower SAE reconstruction MSE at fixed TopK sparsity (k=8,16,32), read as a proxy for less superposition. Gorton and Lewis supply the evidence by pairing a toy-model setup in which superposition is exactly measurable and controllable via sparsity with SAE loss as a proxy in ResNet18. Geometry supports the mechanism: non-robust toy models show about 2× the mean off-diagonal interference (WᵀW) of robust ones, and PGD attacks raise SAE L0 at layer4 from 126 to 351 (2.8×), whereas FGSM barely moves it (1.02×). Robust models also look more robust than their superposition level predicts, which suggests interference structure matters beyond its total amount. The causal claim in real models is only one-directional, because superposition cannot be intervened on there. The paper argues superposition is at least one causal factor and possibly the primary one. If so, robustness trades off against the capacity superposition provides, and interpretability and robustness research share objectives.

What to take away

  1. 1. In toy models with n=100 features compressed into m=20 hidden units, vulnerability to a noise-plus-one-step-gradient L2 attack (ε=10% of mean input norm) rises with feature sparsity and with features per dimension.
  2. 2. Under Elhage-style analytic attacks, features per dimension correlates with adversarial vulnerability at r≈0.99 (p<0.0001) across 30 toy models trained at exponentially spaced densities from 1.0 to 0.1.
  3. 3. Adversarially training toy models (α=0.5 clean/adversarial mix, ε=0.1‖x‖₂, 150,000 steps, lr 1e-3) reduces features per dimension at a given sparsity, which the authors describe as the first demonstration of causality from robustness to superposition.
  4. 4. Non-robust toy models have about 2× the mean off-diagonal interference in WᵀW of robust models trained at the same sparsity, and adversarial inputs light up many interfering features in the superposition graph while clean inputs light up few.
  5. 5. On ImageNet ResNet18s from Salman et al. (2020) with L2 training ε∈{0,1,3,5}, TopK SAEs with k∈{8,16,32} reach lower reconstruction MSE on more robust models at fixed sparsity, which the paper reads as evidence that robustness reduces superposition.
  6. 6. PGD adversarial examples raise SAE L0 at ResNet18 layer4 from 126.0 to 351.4 (2.8×) on n=100,000 images, whereas FGSM barely changes it (128.1, about 1.02×), consistent with iterative attacks accumulating interference.
  7. 7. Methodology another lab could replicate: use feature sparsity as an instrumental variable to set superposition, then confirm the manipulation works by checking features per dimension before reading off robustness, and cross-check with adversarially trained models, which shift superposition independently of the data distribution.
  8. 8. The toy-model vulnerability curve has an unexplained dip at antipodal superposition (phase transitions near 1.7 and 4 on the 1/(1−S) axis), where vulnerability drops temporarily.
  9. 9. Open question: robust toy models are more robust than their total superposition level predicts, and the authors conjecture that the structure of the interference matrix, not only its overall level, determines vulnerability.
  10. 10. If superposition is a primary cause, then robustness necessarily costs the capacity superposition provides, so the paper predicts an unavoidable trade-off and an alignment between interpretability and robustness goals.

Peer brief — for seminar discussion

This work tests Elhage et al.'s 2022 conjecture that superposition drives adversarial examples. The method is a bidirectional intervention design. In a toy ReLU autoencoder with 100 features and 20 hidden units, superposition is exactly measurable as features per dimension and controllable through feature sparsity. In ResNet18 it is not measurable, so the work substitutes sparse-autoencoder reconstruction loss as a proxy. The headline results are four. Vulnerability to a noise-plus-one-step L2 gradient attack rises with sparsity and superposition; Elhage-style analytic attacks give r≈0.99 across 30 models. Adversarial training (α=0.5, 150k steps) lowers features per dimension at matched sparsity. Non-robust toy models carry about 2× the off-diagonal interference of robust ones. Robust ImageNet ResNet18s from Salman et al. (L2 ε up to 5) get lower TopK SAE MSE at k=8, 16 and 32. A further analysis shows PGD raising SAE L0 at layer4 from 126 to 351 (2.8×), while FGSM changes it by about 1.02×. The load-bearing claim is that superposition is at least one causal factor. In real networks the causal arrow is shown only from robustness to the proxy, never from superposition to robustness. The implication is a capacity-robustness trade-off, with interpretability and robustness research pulling in the same direction. A natural alternative to the SAE-loss proxy would be measuring feature interference directly, for instance through decoder-direction coherence, or using circuit-level ablations. The unexplained antipodal dip and the finding that robust models beat their superposition level together suggest interference structure matters more than a scalar count. The strongest pushback is on the real-model evidence. SAE reconstruction loss depends on more than superposition. Adversarial training changes activation statistics, smoothness and effective dimensionality, and plausibly changes feature density and importance, any of which could lower MSE without reducing superposition. Standardizing activations per model does not resolve that. The text says it has no hypothesis for a competing explanation, which is an absence of imagination rather than evidence. The toy-model analysis also uses uniform feature importance, and the toy attack is one-step. The paper's argument that data-distribution confounds bias against its trend is asserted rather than shown. The paper's hypothesis is that superposition is a major, possibly primary, cause. The data fit that claim but cannot rank it against other factors.

Methods (11)

Frameworks (5)

Datasets (2)

Original abstract (expand)

Adversarial examples -- inputs with imperceptible perturbations that fool neural networks -- remain one of deep learning's most perplexing phenomena despite nearly a decade of research. While numerous defenses and explanations have been proposed, there is no consensus on the fundamental mechanism. One underexplored hypothesis is that superposition, a concept from mechanistic interpretability, may be a major contributing factor, or even the primary cause. We present four lines of evidence in support of this hypothesis, greatly extending prior arguments by Elhage et al. (2022): (1) superposition can theoretically explain a range of adversarial phenomena, (2) in toy models, intervening on superposition controls robustness, (3) in toy models, intervening on robustness (via adversarial training) controls superposition, and (4) in ResNet18, intervening on robustness (via adversarial training) controls superposition.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+26 more

Similar preprints — Semantic Scholar