paper
active
2026
paper:doi-10-48550-arxiv-2604-28119

Do Sparse Autoencoders Capture Concept Manifolds?

TL;DR

Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept manifolds as compact subspaces; they spread them across support sets well beyond the manifold's ambient dimension, a fragmented regime the paper calls dilution. The theory separates two ways an SAE can capture a manifold: globally, with a small atom group whose span contains it (Theorem 1, requiring μ < 1/(2k−1) incoherence), or locally, by tiling it with localized features. Tiling splits into shattering (mutually exclusive atoms) and dilution (redundant, mixed-selectivity atoms). A synthetic benchmark of 48 manifold instances (8 types, embedded in R^128, dictionary c=512) shows a sparsity sweet spot near k=4 with shattering below it and dilution above. On Llama, restricted R² plateaus past the ambient dimension and features show tuning curves, for example ones-digit and decade features on the years manifold. The instrument is an Ising-coupling pipeline: a pairwise Ising model fit on binarized codes, then Louvain communities and a PCA spectral-gap check. It recovers temperature, colors and political bias and surfaces a new epistemic-uncertainty manifold, while decoder cosine and Pearson correlation fail. Low reconstruction error therefore does not mean coherent geometry, and single-feature interpretation, steering and auto-interp miss it. The paper argues the unit of interpretability should be geometric objects, meaning groups of atoms, and that featurizers should target manifolds directly.

What to take away

  1. 1. On Llama3.1-8B layer 19, restricted-R² curves averaged over manifolds and architectures plateau at a support size well beyond each manifold's ambient dimension, which places all 25 retained SAEs (VE > 0.85) in the dilution regime instead of compact capture.
  2. 2. PCA of last-token Llama3.1-8B activations shows smooth nonlinear geometry for days (circle), colors (paraboloid), years (helix), temperature, age and formality, and interpolating between PCA centroids shifts next-token probabilities smoothly, for example Wednesday to Thursday.
  3. 3. Theorem 1 gives a subspace-recovery guarantee: if a k-atom group spans the manifold's k-dimensional subspace and the dictionary is μ-incoherent with μ < 1/(2k−1), a sparse decoder captures the manifold at precision O(λ). It is proved for an idealized OMP encoder, not a trained one.
  4. 4. In the synthetic benchmark (48 manifold instances, R^128, c=512 TopK SAEs, k from 3 to 25), capture peaks near k=4 and degrades on both sides: low k shatters manifolds across unrelated atom groups, and high k dilutes them across redundant atoms.
  5. 5. SAE features act as localized detectors with tuning curves. On the years manifold, most architectures learn features that fire periodically every 10 years (the ones digit) alongside decade features, which is the population-code picture from neuroscience.
  6. 6. Ising couplings (and conditional co-activation) give clean block-diagonal affinity structure over supervised top-3 features for colors, days and temperature, while decoder cosine similarity and Pearson correlation do not, so manifold membership is a functional property (which atoms co-fire) and not a geometric one.
  7. 7. Applied to a BatchTopK SAE (×8, k=64) on Llama3.1-8B, the Ising pipeline recovers temperature, colors and political bias as distinct communities and finds a previously unidentified manifold encoding measurement uncertainty in scientific text (+/- statistics, error margins).
  8. 8. Replicable method: fit a pseudo-likelihood Ising model on binarized SAE codes with L-BFGS and a symmetrized J, pick regularization by EBIC (γ=0.5, IsingFit), run Louvain on |J|, and validate each community by a sharp PCA spectral gap in its code vectors.
  9. 9. Optimal-transport comparison across SAEs finds weak alignment in decoder directions and on random data but strong alignment when restricted to a specific manifold, so individual features are unstable while the induced manifold coordinate system is shared.
  10. 10. Open question and hypothesis: ordered regimes (capture, shattering) should give recoverable communities under spectral clustering of |J| while dilution gives mixed-sign couplings that do not. This predicts that instability across runs, brittle single-feature steering and failed auto-interp all trace to tiling, and that featurizers with manifold-level primitives (not point-based simplex SAEs, which Lemma 1 rules out for factor manifolds) are needed.

Peer brief — for seminar discussion

Bhalla, Fel and colleagues ask whether sparse autoencoders capture the curved, low-dimensional structure that representations seem to have. They first show on Llama3.1-8B layer 19 that days (circle), colors (paraboloid), years (helix), temperature and age have smooth PCA geometry, and that steering between centroids smoothly changes output token probabilities. They then formalize an Additive Mixture of Manifolds (Minkowski sum of immersed submanifolds) and define two capture modes. Global subspace capture means a compact atom group spans the manifold, with a recovery guarantee (Theorem 1, μ < 1/(2k−1) incoherence, O(λ) precision) borrowed from sparse-recovery theory. Local tiling means localized atoms cover regions, either shattered (mutually exclusive) or diluted (redundant, mixed selectivity). The instrument they introduce is a pairwise Ising model on binarized SAE codes, with Louvain communities and a spectral-gap check, as a way to find atom groups without supervision. Alternatives are decoder-cosine clustering or Pearson correlation (the paper shows both fail), or Gaussian graphical models like the graphical lasso. The load-bearing finding is that trained SAEs sit in dilution. A synthetic benchmark (48 manifolds in R^128, c=512 TopK) shows a sparsity sweet spot near k=4, with shattering below and dilution above. On Llama, 25 SAEs across five architectures (VE > 0.85, 500M Pile tokens) show restricted-R² curves that plateau far past the ambient dimension, plus tuning curves such as ones-digit and decade features for years. Ising couplings recover temperature, colors and political bias and flag a new epistemic-uncertainty cluster. The implication is that reconstruction quality says little about interpretability, and that single-feature inspection, steering and auto-interp are structurally limited. The paper hypothesizes this also explains dictionary instability across runs. Interpretability should treat groups of atoms or geometric objects as the unit. A critical reader would push on scope and evidence. The headline dilution claim rests on one model, one layer and hand-chosen template-built manifolds. Capture is judged against an ambient dimension that the authors define and supervise, and the sweet spot at k=4 is tuned to a synthetic L0=4 generator, so the regime boundaries may be artifacts of the setup. The theorem assumes an idealized OMP encoder and an exactly spanning atom set, which trained SAEs never provide. The explanations for earlier negative SAE results are asserted, not tested, and the novel uncertainty manifold gets only qualitative support, with no steering or causal check.

Methods (10)

Frameworks (8)

Datasets (5)

Findings (19)

Claims (18)

Original abstract (expand)

Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+18 more

Similar preprints — Semantic Scholar

Cited by (1)