paper:doi-10-48550-arxiv-2509-02565Understanding sparse autoencoder scaling in the presence of feature manifolds
TL;DR
Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead of discovering rarer features. Michaud, Gorton and McGrath adapt Brill's (2024) capacity-allocation model to SAEs. The model casts SAE training as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss L_i(n_i) ∝ n_i^-β. It predicts two regimes. When α < β, loss scales as N^-α and discovered features D(N) grow proportionally to N. When β < α, loss scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0. In a simulation where one manifold has β ≪ α, an SAE with 100 million latents discovers only 3 million features. ReLU L1 SAEs on synthetic hyperspheres show β ≈ 0.05 for dimensions 6–8, which is pathological. Once radial variation is added, loss plateaus at roughly n_i ≈ 2d_i latents, a basis solution. Gemma Scope layer-12 latent frequency slopes of −0.57 to −0.74 suggest α ≈ 0.5–0.7. The paper does not settle whether real SAEs are pathological. It argues that the risk depends on manifold geometry, and that realistic radial variation may protect against it. It presents the work as a framing of the problem, not an answer.
What to take away
- 1. The model reduces SAE training to choosing latent counts n_i per feature to minimize Σ p_i L_i(n_i) subject to Σ n_i = N, which assumes feature-specific latents and orthogonal feature subspaces.
- 2. With purely discrete features and p_i ∝ i^-(1+α), the optimal SAE gives one latent to each of the N most frequent features, so D(N) = N and loss falls as N^-α, which matches Anthropic's finding that SAEs learn features roughly in frequency order.
- 3. The pathological threshold is β < α: loss then scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0, whereas α < β gives D(N) ∝ N and loss N^-α.
- 4. In a simulation where only the most frequent feature is a manifold with L(n) ∝ n^-β and all others are discrete, with p_i ∝ i^-1.5, a simulated SAE with 100 million latents discovers only about 3 million features when β ≪ α.
- 5. ReLU L1 SAEs trained on a circle S^1 in R^2 reduce loss by tiling it with more sparsely firing latents (4, 8 and 24 latents shown), and the manifold accommodates dozens of latents before the loss plateaus.
- 6. On hollow hyperspheres, higher-dimensional cases (dimension 6–8) keep reducing loss out to at least 10^4 latents, with a measured β of roughly 0.05 between 10^2 and 10^4 latents, far below plausible α values.
- 7. When toy manifolds have radial variation (points sampled with 0.5 < |x| < 2), both ReLU and JumpReLU SAEs plateau at about n_i ≈ 2d_i latents, which suggests a positive/negative basis solution, and the paper calls this the strongest argument against pathological scaling.
- 8. Gemma Scope layer-12 latent firing frequencies sorted by rank have slopes of −0.74 (16k latents) and −0.57 (32k latents) between latent indices 10^2 and 10^4, which implies α ≈ 0.5–0.7 if latents map one-to-one to features.
- 9. Language-model scaling exponents of 0.07 (Kaplan) and 0.34 (Hoffmann), read through the quantization model, would put α in that same range, though the paper calls this highly speculative.
- 10. Open question: in gemma-2-2b layer 12 with a 262k-width SAE, 10–100 latents have decoder cosine similarity >0.97 with their nearest neighbor, but each of those neighbors is dead over 250M tokens, and many alive ones fire on single digits or letters, so it is unclear whether an earlier manifold tiling was pruned by L0 pressure.
Peer brief — for seminar discussion
Michaud, Gorton and McGrath ask whether multi-dimensional feature manifolds can make SAE scaling pathological. They adapt Brill's (2024) capacity-allocation model, written for neural scaling, to SAEs. Training is treated as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss curves L_i(n_i) ∝ n_i^-β. The assumptions are that latents are feature-specific, decoder directions lie in the feature's subspace, and feature subspaces are orthogonal. The load-bearing result is a phase transition at α = β. When α < β, loss scales as N^-α and discovered features D(N) grow linearly in N, as in the discrete case. When β < α, loss scales as N^-β, D(N) ∝ N^((1+β)/(1+α)), and D(N)/N goes to 0. A numerical simulation with one slowly improving manifold and p_i ∝ i^-1.5 has a 100M-latent SAE discover only about 3M features. Empirically, the evidence is mixed. ReLU L1 SAEs on hollow hyperspheres of dimension 6–8 give β ≈ 0.05 between 10^2 and 10^4 latents, which is pathological. Adding radial variation makes the loss plateau at about 2d_i latents for both ReLU and JumpReLU SAEs, a basis solution. Gemma Scope layer-12 firing frequencies have slopes of −0.57 to −0.74, which would suggest α ≈ 0.5–0.7. The implication is that manifold-driven dark matter is plausible but depends on geometry. Radial variation, which the paper cites as seen in Engels et al., may make the problem go away. The paper itself calls this work framing, not resolution. A critical reader would push on the idealized assumptions. Orthogonal subspaces, feature-specific latents and no absorption are violated in practice, as the paper concedes. Estimating α from latent firing frequencies conflates true feature frequencies with absorption, composition and tiling, and the neural-scaling-exponent route, spanning 0.07 to 0.34, is too loose to constrain anything. β comes only from synthetic geometries, so the conclusion depends on which toy shape you pick. An alternative would have been to measure per-feature loss curves directly on real activations, for example by training SAEs of several widths on the same layer and tracking how decoder clusters grow. The decoder cosine-similarity analysis is a weak proxy for that. In Inception-v1, high-similarity pairs may just be duplicated latents.
Methods (6)
- Decoder nearest-neighbor cosine similarity analysisDetects tiling in real SAEs.
- Lagrange multiplier continuous-relaxation derivationDerives regimes.
- Latent firing frequency slope measurementEstimates α.
- Numerical simulation of heterogeneous latent allocationSimulates one power-law manifold among discrete features.
- ReLU SAE with L1 penaltyTrained on toy manifolds.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
Frameworks (6)
- Brill capacity-allocation modelModels allocate capacity among power-law-distributed data manifolds.
- Feature ManifoldsHypothesized extension of superposition where features may be higher-dimensional manifolds rather than 1D directions
- JumpReLU SAEEvaluated architecture.
- Multi-dimensional linear representation hypothesisActivations are sums of sparse features living in subspaces.
- Quantization (quanta) model of neural scalingFeatures analogous to quanta.
- SAE latent allocation modelReduces SAE optimization to allocating N latents across features.
Datasets (3)
- Gemma Scope SAEs on gemma-2-2b layer 12Real SAEs analyzed.
- Inception-v1 mixed3b activations and SAEsVision SAEs analyzed.
- Synthetic toy manifolds (hyperspheres and spherical shells)Training data for L(n) curves.
Findings (13)
- Gemma Scope decoder nearest-neighbor cosine similarities skewed above random but few extremely high
Fig 7.
- Higher-dimensional hyperspheres accommodate ≥10^4 latents without plateau; β≈0.05 for dim 6-8
Fig 5.
- Inception-v1 SAE shows meaningful fraction of latents with very high nearest-neighbor cosine
Fig 8; may be duplication.
- In one 262k Gemma Scope SAE, 10-100 latents with cosine>0.97 had dead nearest neighbors
Training artifact.
- Language model parameter scaling exponents 0.07 (Kaplan) and 0.34 (Hoffmann)
Speculative α estimate.
- With radial variation, ReLU SAE manifolds plateau early near n≈2d_i latents
Strongest argument against pathology.
- ReLU SAEs reduce loss slightly by tiling circle S^1 with 4, 8, 24 latents
Fig 1.
- Gemma Scope layer 12 latent frequency slopes -0.74 (16k) and -0.57 (32k)
Implies α≈0.5-0.7.
- JumpReLU SAEs also plateau after ~2d_i latents with radial variation
Fig 6.
- Simulated SAE with 100M latents discovers only ~3M features when β≪α
Fig 2 right.
Claims (9)
- Feature absorption and non-orthogonality violate the model's assumptions but are accepted for expedience
Limitation.
- Radial variation in realistic manifolds is the strongest argument against pathological manifold scaling
Appendix A.1.
- When β<α, SAEs can optimally tile common feature manifolds at the expense of discovering rarer features
Core concern.
- Saturation at n≈2d_i likely reflects a basis solution with positive and negative latents per direction
Explains plateau.
- The SAE scaling model mirrors the quanta model of neural scaling with features as quanta
Link to prior work.
- Scaling SAEs on discrete features learns more features in decreasing order of frequency
Discrete case.
- The work is primarily about framing rather than answering the problem
Authors' stance.
- Whether real SAEs are in the pathological regime remains unresolved
Uncertain α and β.
- The L(n) curve shape depends on manifold geometry
Synthetic experiments.
Hypotheses (2)
Questions (2)
Original abstract (expand)
Sparse autoencoders (SAEs) model the activations of a neural network as linear combinations of sparsely occurring directions of variation (latents). The ability of SAEs to reconstruct activations follows scaling laws w.r.t. the number of latents. In this work, we adapt a capacity-allocation model from the neural scaling literature (Brill, 2024) to understand SAE scaling, and in particular, to understand how "feature manifolds" (multi-dimensional features) influence scaling behavior. Consistent with prior work, the model recovers distinct scaling regimes. Notably, in one regime, feature manifolds have the pathological effect of causing SAEs to learn far fewer features in data than there are latents in the SAE. We provide some preliminary discussion on whether or not SAEs are in this pathological regime in the wild.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 85%
- The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling LawsMaciej Trzaskowski, Quan Nguyen, Fred Roosta Eslam Zaher2026≈ 87%
- Sparse Autoencoder Features for Classifications and TransferabilityShan Chen, Kuleen Sasse, Hugo Aerts, Thomas Hartvigsen, Danielle S. Bitterman Jack Gallifant2026≈ 86%
- Sparse Semantic Dimension as a Generalization Certificate for LLMsAsif Ekbal Dibyanayan Bandyopadhyay2026≈ 85%
- Sparse Autoencoder Decomposition of Clinical Sequence Model Representations: Feature Complexity, Task Specialisation, and Mortality PredictionFeng Dong, Andreas Karwath Chris Sainsbury2026≈ 85%
- Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse AutoencodersMudith Jayasekara, Max Kirkby Charles O'Neill2025≈ 85%
- Do Sparse Autoencoders Capture Concept Manifolds?in corpus2026≈ 85%
- Improving Robustness In Sparse Autoencoders via Masked RegularizationKowshik Thopalli, Bhavya Kailkhura and Wesam Sakla Vivek Narayanaswamy2026≈ 85%
- Evaluating and Designing Sparse Autoencoders by Approximating Quasi-OrthogonalityAdam Davies, Marc E. Canby and Julia Hockenmaier Sewoong Lee2025≈ 84%
- Toy Models of Superpositioncitedin corpus2022≈ 79%
- How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language ModelsElies Segu\'i-Mas, Guillermina Tormo-Carb\'o Hector Borobia2026≈ 84%
- SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMsAndrew Gordon, Peter Lai, Xavier Poncini, David Quarel, and Stefan Heimersheim Sean P. Fillingham2025≈ 84%
- Improving Sparse Autoencoder with Dynamic AttentionJinsen Zhang, Dawei Su, Hui Huang Dongsheng Wang2026≈ 84%
- Learning biologically relevant features in a pathology foundation model using sparse autoencodersCiyue Shen, Neel Patel, Chintan Shah, Darpan Sanghavi, Blake Martin, Alfred Eng, Daniel Shenker, Harshith Padigela, Raymond Biju, Syed Ashar Javed, Jennifer Hipp, John Abel, Harsha Pokkalla, Sean Grullon, Dinkar Juyal Nhat Minh Le2024≈ 84%
- ≈ 84%
- Geometric regularization of autoencoders via observed stochastic dynamicsFelix X.-F. Ye Sean Hill2026≈ 84%
- ≈ 84%
- Evaluating Synthetic Activations composed of SAE Latents in GPT-2Nora Petrova, Chatrik Singh Mangat, Jett Janiak, Stefan Heimersheim Giorgi Giglemiani2024≈ 84%
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentionin corpus2026≈ 82%
- ≈ 82%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 82%
- ≈ 82%
- ≈ 81%
- ≈ 80%
- Stochastic Parameter Decompositionin corpus2025≈ 80%
- ≈ 79%
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilityin corpus2025≈ 79%
- ≈ 79%
- ≈ 79%
- ≈ 75%
+25 more
Similar preprints — Semantic Scholar
Cited by (4)
- Do Sparse Autoencoders Capture Concept Manifolds?
Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Larger models learn tasks smaller models never learn, and the cause is data-induced competition for neurons, not expressivity: small models spend capacity on frequent or simple tasks, and their update
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse
- When Models Manipulate Manifolds: The Geometry of a Counting Task
Claude 3.5 Haiku's linebreaking in fixed-width text is implemented by manipulating low-dimensional, rippled 1-D feature manifolds: character count, line width, characters remaining and next-token leng