thinker
active
thinker:gorton-liv

Gorton, Liv

Authored
2
Introduces
0
Studies
0
Affiliations
0
Cited by
4

Authored papers (2)

  • Superposition causally contributes to adversarial vulnerability, and the link runs in both directions: in a toy ReLU autoencoder (n=100 features, m=20 hidden units), raising feature sparsity raises features-per-dimension and adversarial vulnerability (r≈0.99 under Elhage-style attacks), and adversarial training with α=0.5, ε=0.1‖x‖₂ for 150,000 steps lowers features-per-dimension at matched sparsity. The same direction holds in real networks: robust ImageNet ResNet18s from Salman et al. (2020) trained at L2 ε∈{0,1,3,5} give lower SAE reconstruction MSE at fixed TopK sparsity (k=8,16,32), read as a proxy for less superposition. Gorton and Lewis supply the evidence by pairing a toy-model setup in which superposition is exactly measurable and controllable via sparsity with SAE loss as a proxy in ResNet18. Geometry supports the mechanism: non-robust toy models show about 2× the mean off-diagonal interference (WᵀW) of robust ones, and PGD attacks raise SAE L0 at layer4 from 126 to 351 (2.8×), whereas FGSM barely moves it (1.02×). Robust models also look more robust than their superposition level predicts, which suggests interference structure matters beyond its total amount. The causal claim in real models is only one-directional, because superposition cannot be intervened on there. The paper argues superposition is at least one causal factor and possibly the primary one. If so, robustness trades off against the capacity superposition provides, and interpretability and robustness research share objectives.

  • Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead of discovering rarer features. Michaud, Gorton and McGrath adapt Brill's (2024) capacity-allocation model to SAEs. The model casts SAE training as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss L_i(n_i) ∝ n_i^-β. It predicts two regimes. When α < β, loss scales as N^-α and discovered features D(N) grow proportionally to N. When β < α, loss scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0. In a simulation where one manifold has β ≪ α, an SAE with 100 million latents discovers only 3 million features. ReLU L1 SAEs on synthetic hyperspheres show β ≈ 0.05 for dimensions 6–8, which is pathological. Once radial variation is added, loss plateaus at roughly n_i ≈ 2d_i latents, a basis solution. Gemma Scope layer-12 latent frequency slopes of −0.57 to −0.74 suggest α ≈ 0.5–0.7. The paper does not settle whether real SAEs are pathological. It argues that the risk depends on manifold geometry, and that realistic radial variation may protect against it. It presents the work as a framing of the problem, not an answer.

More papers — OpenAlex / S2

Co-authors (12)