paper
active
2025
paper:doi-10-48550-arxiv-2509-02565

Understanding sparse autoencoder scaling in the presence of feature manifolds

TL;DR

Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead of discovering rarer features. Michaud, Gorton and McGrath adapt Brill's (2024) capacity-allocation model to SAEs. The model casts SAE training as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss L_i(n_i) ∝ n_i^-β. It predicts two regimes. When α < β, loss scales as N^-α and discovered features D(N) grow proportionally to N. When β < α, loss scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0. In a simulation where one manifold has β ≪ α, an SAE with 100 million latents discovers only 3 million features. ReLU L1 SAEs on synthetic hyperspheres show β ≈ 0.05 for dimensions 6–8, which is pathological. Once radial variation is added, loss plateaus at roughly n_i ≈ 2d_i latents, a basis solution. Gemma Scope layer-12 latent frequency slopes of −0.57 to −0.74 suggest α ≈ 0.5–0.7. The paper does not settle whether real SAEs are pathological. It argues that the risk depends on manifold geometry, and that realistic radial variation may protect against it. It presents the work as a framing of the problem, not an answer.

What to take away

  1. 1. The model reduces SAE training to choosing latent counts n_i per feature to minimize Σ p_i L_i(n_i) subject to Σ n_i = N, which assumes feature-specific latents and orthogonal feature subspaces.
  2. 2. With purely discrete features and p_i ∝ i^-(1+α), the optimal SAE gives one latent to each of the N most frequent features, so D(N) = N and loss falls as N^-α, which matches Anthropic's finding that SAEs learn features roughly in frequency order.
  3. 3. The pathological threshold is β < α: loss then scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0, whereas α < β gives D(N) ∝ N and loss N^-α.
  4. 4. In a simulation where only the most frequent feature is a manifold with L(n) ∝ n^-β and all others are discrete, with p_i ∝ i^-1.5, a simulated SAE with 100 million latents discovers only about 3 million features when β ≪ α.
  5. 5. ReLU L1 SAEs trained on a circle S^1 in R^2 reduce loss by tiling it with more sparsely firing latents (4, 8 and 24 latents shown), and the manifold accommodates dozens of latents before the loss plateaus.
  6. 6. On hollow hyperspheres, higher-dimensional cases (dimension 6–8) keep reducing loss out to at least 10^4 latents, with a measured β of roughly 0.05 between 10^2 and 10^4 latents, far below plausible α values.
  7. 7. When toy manifolds have radial variation (points sampled with 0.5 < |x| < 2), both ReLU and JumpReLU SAEs plateau at about n_i ≈ 2d_i latents, which suggests a positive/negative basis solution, and the paper calls this the strongest argument against pathological scaling.
  8. 8. Gemma Scope layer-12 latent firing frequencies sorted by rank have slopes of −0.74 (16k latents) and −0.57 (32k latents) between latent indices 10^2 and 10^4, which implies α ≈ 0.5–0.7 if latents map one-to-one to features.
  9. 9. Language-model scaling exponents of 0.07 (Kaplan) and 0.34 (Hoffmann), read through the quantization model, would put α in that same range, though the paper calls this highly speculative.
  10. 10. Open question: in gemma-2-2b layer 12 with a 262k-width SAE, 10–100 latents have decoder cosine similarity >0.97 with their nearest neighbor, but each of those neighbors is dead over 250M tokens, and many alive ones fire on single digits or letters, so it is unclear whether an earlier manifold tiling was pruned by L0 pressure.

Peer brief — for seminar discussion

Michaud, Gorton and McGrath ask whether multi-dimensional feature manifolds can make SAE scaling pathological. They adapt Brill's (2024) capacity-allocation model, written for neural scaling, to SAEs. Training is treated as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss curves L_i(n_i) ∝ n_i^-β. The assumptions are that latents are feature-specific, decoder directions lie in the feature's subspace, and feature subspaces are orthogonal. The load-bearing result is a phase transition at α = β. When α < β, loss scales as N^-α and discovered features D(N) grow linearly in N, as in the discrete case. When β < α, loss scales as N^-β, D(N) ∝ N^((1+β)/(1+α)), and D(N)/N goes to 0. A numerical simulation with one slowly improving manifold and p_i ∝ i^-1.5 has a 100M-latent SAE discover only about 3M features. Empirically, the evidence is mixed. ReLU L1 SAEs on hollow hyperspheres of dimension 6–8 give β ≈ 0.05 between 10^2 and 10^4 latents, which is pathological. Adding radial variation makes the loss plateau at about 2d_i latents for both ReLU and JumpReLU SAEs, a basis solution. Gemma Scope layer-12 firing frequencies have slopes of −0.57 to −0.74, which would suggest α ≈ 0.5–0.7. The implication is that manifold-driven dark matter is plausible but depends on geometry. Radial variation, which the paper cites as seen in Engels et al., may make the problem go away. The paper itself calls this work framing, not resolution. A critical reader would push on the idealized assumptions. Orthogonal subspaces, feature-specific latents and no absorption are violated in practice, as the paper concedes. Estimating α from latent firing frequencies conflates true feature frequencies with absorption, composition and tiling, and the neural-scaling-exponent route, spanning 0.07 to 0.34, is too loose to constrain anything. β comes only from synthetic geometries, so the conclusion depends on which toy shape you pick. An alternative would have been to measure per-feature loss curves directly on real activations, for example by training SAEs of several widths on the same layer and tracking how decoder clusters grow. The decoder cosine-similarity analysis is a weak proxy for that. In Inception-v1, high-similarity pairs may just be duplicated latents.

Methods (6)

Frameworks (6)

Original abstract (expand)

Sparse autoencoders (SAEs) model the activations of a neural network as linear combinations of sparsely occurring directions of variation (latents). The ability of SAEs to reconstruct activations follows scaling laws w.r.t. the number of latents. In this work, we adapt a capacity-allocation model from the neural scaling literature (Brill, 2024) to understand SAE scaling, and in particular, to understand how "feature manifolds" (multi-dimensional features) influence scaling behavior. Consistent with prior work, the model recovers distinct scaling regimes. Notably, in one regime, feature manifolds have the pathological effect of causing SAEs to learn far fewer features in data than there are latents in the SAE. We provide some preliminary discussion on whether or not SAEs are in this pathological regime in the wild.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+25 more

Similar preprints — Semantic Scholar

Cited by (4)