thinker:mcgrath-tomMcGrath, Tom
Authored papers (1)
Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead of discovering rarer features. Michaud, Gorton and McGrath adapt Brill's (2024) capacity-allocation model to SAEs. The model casts SAE training as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss L_i(n_i) ∝ n_i^-β. It predicts two regimes. When α < β, loss scales as N^-α and discovered features D(N) grow proportionally to N. When β < α, loss scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0. In a simulation where one manifold has β ≪ α, an SAE with 100 million latents discovers only 3 million features. ReLU L1 SAEs on synthetic hyperspheres show β ≈ 0.05 for dimensions 6–8, which is pathological. Once radial variation is added, loss plateaus at roughly n_i ≈ 2d_i latents, a basis solution. Gemma Scope layer-12 latent frequency slopes of −0.57 to −0.74 suggest α ≈ 0.5–0.7. The paper does not settle whether real SAEs are pathological. It argues that the risk depends on manifold geometry, and that realistic radial variation may protect against it. It presents the work as a framing of the problem, not an answer.
More papers — OpenAlex / S2
Co-authors (8)
- Gorton, Liv4 shared
- Michaud, Eric J.4 shared
- Ari Brill2 shared
- Chris Olah2 shared
- Eric Michaud2 shared
- Liv Gorton2 shared
- Neel Nanda2 shared
- Tom McGrath2 shared
Their work is cited by (4)
- When Models Manipulate Manifolds: The Geometry of a Counting Task2× refs
- Do Sparse Autoencoders Capture Concept Manifolds?2× refs
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention2× refs
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds2× refs