paper:doi-10-48550-arxiv-2604-28119Do Sparse Autoencoders Capture Concept Manifolds?
TL;DR
Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept manifolds as compact subspaces; they spread them across support sets well beyond the manifold's ambient dimension, a fragmented regime the paper calls dilution. The theory separates two ways an SAE can capture a manifold: globally, with a small atom group whose span contains it (Theorem 1, requiring μ < 1/(2k−1) incoherence), or locally, by tiling it with localized features. Tiling splits into shattering (mutually exclusive atoms) and dilution (redundant, mixed-selectivity atoms). A synthetic benchmark of 48 manifold instances (8 types, embedded in R^128, dictionary c=512) shows a sparsity sweet spot near k=4 with shattering below it and dilution above. On Llama, restricted R² plateaus past the ambient dimension and features show tuning curves, for example ones-digit and decade features on the years manifold. The instrument is an Ising-coupling pipeline: a pairwise Ising model fit on binarized codes, then Louvain communities and a PCA spectral-gap check. It recovers temperature, colors and political bias and surfaces a new epistemic-uncertainty manifold, while decoder cosine and Pearson correlation fail. Low reconstruction error therefore does not mean coherent geometry, and single-feature interpretation, steering and auto-interp miss it. The paper argues the unit of interpretability should be geometric objects, meaning groups of atoms, and that featurizers should target manifolds directly.
What to take away
- 1. On Llama3.1-8B layer 19, restricted-R² curves averaged over manifolds and architectures plateau at a support size well beyond each manifold's ambient dimension, which places all 25 retained SAEs (VE > 0.85) in the dilution regime instead of compact capture.
- 2. PCA of last-token Llama3.1-8B activations shows smooth nonlinear geometry for days (circle), colors (paraboloid), years (helix), temperature, age and formality, and interpolating between PCA centroids shifts next-token probabilities smoothly, for example Wednesday to Thursday.
- 3. Theorem 1 gives a subspace-recovery guarantee: if a k-atom group spans the manifold's k-dimensional subspace and the dictionary is μ-incoherent with μ < 1/(2k−1), a sparse decoder captures the manifold at precision O(λ). It is proved for an idealized OMP encoder, not a trained one.
- 4. In the synthetic benchmark (48 manifold instances, R^128, c=512 TopK SAEs, k from 3 to 25), capture peaks near k=4 and degrades on both sides: low k shatters manifolds across unrelated atom groups, and high k dilutes them across redundant atoms.
- 5. SAE features act as localized detectors with tuning curves. On the years manifold, most architectures learn features that fire periodically every 10 years (the ones digit) alongside decade features, which is the population-code picture from neuroscience.
- 6. Ising couplings (and conditional co-activation) give clean block-diagonal affinity structure over supervised top-3 features for colors, days and temperature, while decoder cosine similarity and Pearson correlation do not, so manifold membership is a functional property (which atoms co-fire) and not a geometric one.
- 7. Applied to a BatchTopK SAE (×8, k=64) on Llama3.1-8B, the Ising pipeline recovers temperature, colors and political bias as distinct communities and finds a previously unidentified manifold encoding measurement uncertainty in scientific text (+/- statistics, error margins).
- 8. Replicable method: fit a pseudo-likelihood Ising model on binarized SAE codes with L-BFGS and a symmetrized J, pick regularization by EBIC (γ=0.5, IsingFit), run Louvain on |J|, and validate each community by a sharp PCA spectral gap in its code vectors.
- 9. Optimal-transport comparison across SAEs finds weak alignment in decoder directions and on random data but strong alignment when restricted to a specific manifold, so individual features are unstable while the induced manifold coordinate system is shared.
- 10. Open question and hypothesis: ordered regimes (capture, shattering) should give recoverable communities under spectral clustering of |J| while dilution gives mixed-sign couplings that do not. This predicts that instability across runs, brittle single-feature steering and failed auto-interp all trace to tiling, and that featurizers with manifold-level primitives (not point-based simplex SAEs, which Lemma 1 rules out for factor manifolds) are needed.
Peer brief — for seminar discussion
Bhalla, Fel and colleagues ask whether sparse autoencoders capture the curved, low-dimensional structure that representations seem to have. They first show on Llama3.1-8B layer 19 that days (circle), colors (paraboloid), years (helix), temperature and age have smooth PCA geometry, and that steering between centroids smoothly changes output token probabilities. They then formalize an Additive Mixture of Manifolds (Minkowski sum of immersed submanifolds) and define two capture modes. Global subspace capture means a compact atom group spans the manifold, with a recovery guarantee (Theorem 1, μ < 1/(2k−1) incoherence, O(λ) precision) borrowed from sparse-recovery theory. Local tiling means localized atoms cover regions, either shattered (mutually exclusive) or diluted (redundant, mixed selectivity). The instrument they introduce is a pairwise Ising model on binarized SAE codes, with Louvain communities and a spectral-gap check, as a way to find atom groups without supervision. Alternatives are decoder-cosine clustering or Pearson correlation (the paper shows both fail), or Gaussian graphical models like the graphical lasso. The load-bearing finding is that trained SAEs sit in dilution. A synthetic benchmark (48 manifolds in R^128, c=512 TopK) shows a sparsity sweet spot near k=4, with shattering below and dilution above. On Llama, 25 SAEs across five architectures (VE > 0.85, 500M Pile tokens) show restricted-R² curves that plateau far past the ambient dimension, plus tuning curves such as ones-digit and decade features for years. Ising couplings recover temperature, colors and political bias and flag a new epistemic-uncertainty cluster. The implication is that reconstruction quality says little about interpretability, and that single-feature inspection, steering and auto-interp are structurally limited. The paper hypothesizes this also explains dictionary instability across runs. Interpretability should treat groups of atoms or geometric objects as the unit. A critical reader would push on scope and evidence. The headline dilution claim rests on one model, one layer and hand-chosen template-built manifolds. Capture is judged against an ambient dimension that the authors define and supervise, and the sweet spot at k=4 is tuned to a synthetic L0=4 generator, so the regime boundaries may be artifacts of the setup. The theorem assumes an idealized OMP encoder and an exactly spanning atom set, which trained SAEs never provide. The explanations for earlier negative SAE results are asserted, not tested, and the novel uncertainty manifold gets only qualitative support, with no steering or causal check.
Methods (10)
- Activation patching along manifold PCA componentsInterpolating along PCA centroids to show causal effect.
- Ising-based unsupervised manifold discovery pipelineFits Ising over binarized codes, Louvain clustering, PCA spectral-gap validation.
- Louvain community detectionPartitions atoms by |J| into manifold groups.
- Matryoshka SAEArchitecture compared.
- Optimal transport comparison of SAE featuresCompares features across SAEs.
- PCA projection of concept activationsUsed to reveal manifold geometry in Llama activations.
- Pseudo-likelihood maximization with EBIC (IsingFit)Procedure for fitting Ising couplings.
- Restricted R² protocolGreedy atom selection measuring variance explained as function of support size.
- sparse autoencodersExisting method for model interpretability that decodes model activations rather than parameters themselves, noted as incomplete solution.
- Standard L1 SAEArchitecture compared.
Frameworks (8)
- Additive Mixture of Manifolds modelData-generating process: activation is sparse sum of low-dimensional manifold points.
- BatchTopK SAEEvaluated architecture.
- Geometric Duality of Sparse ConceptsDistinguishes direction-based (cone) and point-based (convex hull) dictionary learning.
- Ising modelPrototypical spin system; baseline for phase transition arguments
- Ising signatures of capture, shattering, and dilutionDefines regimes via group size and signed cohesion of Ising couplings.
- JumpReLU SAEEvaluated architecture.
- Linear Representation HypothesisThe hypothesis that models internalize concepts as approximately linear directions in representation space; used to interpret MDS injection behavior
- TopK SAEEvaluated architecture (Gao et al.).
Datasets (5)
- Llama-3.1-8B layer-19 activationsRepresentation source for manifold and SAE analyses.
- Manifold concept datasets (colors, days, years, temperature, age, formality, etc.)Template-prompt datasets of continuous concepts.
- Political bias dataset (Jones 2024)Source for political bias manifold.
- Synthetic manifold zoo benchmark48 instances of eight manifold types in R^128 mixed sparsely.
- The Pile (uncopyrighted subset)500M tokens used for SAE training.
Findings (19)
- Ising couplings and conditional co-activation give cleanest block-diagonal affinity for colors, days, temperature; decoder cosine and Pearson fail
Comparison of five similarity metrics.
- Ising pipeline on BatchTopK SAE (layer 19, ×8, k=64) recovers temperature, colors, and political bias manifolds as distinct communities
Unsupervised recovery of known manifolds.
- SAE reconstructions with increasing numbers of features approximate manifolds piecewise-linearly, each feature covering a local patch
Features act as local patch detectors.
- Receptive field plots on days-of-week show selectivity per day and architectural biases (angular separability in TopK, linear in L1)
Ambient-space shattering.
- Restricted-R² variance explained on Llama3.1-8B plateaus well beyond each manifold's ambient dimension across five SAE architectures
SAEs do not achieve compact capture; they are in dilution.
- If D is µ-incoherent, µ<1/(2k−1), and S* spans the subspace, an idealized sparse decoder captures M at precision O(λ) (Theorem 1)
Formal subspace recovery guarantee.
- Pairwise Ising coupling matrix recovers ground-truth manifold block structure on synthetic data even outside capture regime
Post hoc recovery works synthetically.
- PCA of Llama3.1-8B layer-19 activations shows smooth curved geometries for age, temperature, day, color, years, and others
Evidence that manifolds exist in LLM representations.
- Ising pipeline surfaces a novel manifold encoding epistemic uncertainty and measurement error in scientific contexts
Hypothesis-generating discovery.
- Clean Ising block-diagonal structure emerges at intermediate K≈8–16 while low K is weak and high K fragments blocks
Coupling structure across sparsity regimes.
Claims (18)
- Many negative findings about SAEs (instability, brittle steering, hard automated interpretation) reflect a mismatch between direction-based interpretation and nonlinear geometry
Geometric explanation for prior negative results.
- Individual features offer only a narrow view of a concept; only the group of features tiling a manifold carries geometric meaning
Implication for interpretability.
- SAEs can capture manifolds either globally via a compact atom group spanning the manifold or locally via features tiling regions
Theoretical framework answer.
- Future representation learning should treat geometric objects, not individual directions, as the basic units of interpretability
Paper's main prescriptive conclusion.
- SAE atoms remain one-dimensional and the loss rewards reconstruction without rewarding coherent treatment of geometric objects
Source of mismatch.
- Capture yields positive couplings, shattering negative couplings, and dilution mixed couplings in the Ising interaction matrix
Signature of each regime.
- Dilution is the genuinely problematic regime for interpretability because it preserves geometry without coherent organization
Framing in App. G.
- Point-based SAEs tile the joint manifold as a single object and cannot recover factor manifolds under superposition
Justifies focusing on direction-based SAEs.
- Manifold membership is a functional property (which atoms co-activate), not a geometric one (where atoms point)
Why decoder cosine fails.
- Understanding neural networks requires shifting from a dictionary of concepts to a geometry of representations
Closing statement.
Hypotheses (1)
Questions (2)
- Whether real representations satisfy the manifold assumption, and whether "manifold" survives as the right name, remains an open problem
Acknowledged open issue in footnote.
- do SAEs capture such manifolds?
Central research question.
Original abstract (expand)
Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 85%
- When Models Manipulate Manifolds: The Geometry of a Counting Taskcitedin corpus2026≈ 83%
- ≈ 83%
- ≈ 86%
- Step-Level Sparse Autoencoder for Reasoning Process InterpretationJiayu Liu, Yuhang Lai, Hao Xu, Zhenya Huang, Ning Miao Xuan Yang2026≈ 85%
- The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling LawsMaciej Trzaskowski, Quan Nguyen, Fred Roosta Eslam Zaher2026≈ 85%
- Open Problems in Mechanistic Interpretabilitycitedin corpus2025≈ 80%
- Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse AutoencodersMudith Jayasekara, Max Kirkby Charles O'Neill2025≈ 84%
- Sparsification and Reconstruction from the Perspective of Representation GeometryBingzhe Wu, Zhile Yang, Chengke Wu Wenjie Sun2025≈ 84%
- Sparse Autoencoder Features for Classifications and TransferabilityShan Chen, Kuleen Sasse, Hugo Aerts, Thomas Hartvigsen, Danielle S. Bitterman Jack Gallifant2026≈ 84%
- Sparse Semantic Dimension as a Generalization Certificate for LLMsAsif Ekbal Dibyanayan Bandyopadhyay2026≈ 84%
- Supervised sparse auto-encoders for interpretable and compositional representationsHugo Wallner, Yoonsoo Nam, Haixuan Xavier Tao Ouns El Harzli2026≈ 83%
- How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language ModelsElies Segu\'i-Mas, Guillermina Tormo-Carb\'o Hector Borobia2026≈ 83%
- Improving Sparse Autoencoder with Dynamic AttentionJinsen Zhang, Dawei Su, Hui Huang Dongsheng Wang2026≈ 83%
- MetaSAEs: Joint Training with a Decomposability Penalty Produces More Atomic Sparse Autoencoder LatentsMatthew Levinson2026≈ 83%
- ≈ 83%
- Domain-Filtered Knowledge Graphs from Sparse Autoencoder FeaturesAbeynaya Gnanasekaran, Eric Darve John Winnicki2026≈ 83%
- A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language ModelsXuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, Mengnan Du Dong Shu2025≈ 83%
- The Deepfake Detective: Interpreting Neural Forensics Through Sparse Features and ManifoldsJared Junkin Subramanyam Sahoo2025≈ 83%
- A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious MinimaHarshvardhan Saini, Zhaoqian Yao, Zheng Lin, Yizhen Liao, Jingyi Cui, Yisen Wang, Mengnan Du, Dianbo Liu Yiming Tang2026≈ 83%
- Interpreting Attention Layer Outputs with Sparse AutoencodersRobert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, Neel Nanda Connor Kissane2024≈ 83%
- ≈ 82%
- ≈ 81%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 81%
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilityin corpus2025≈ 80%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 80%
- The World Inside Neural Networksin corpus2026≈ 80%
- ≈ 80%
- ≈ 79%
- ≈ 78%
+18 more
Similar preprints — Semantic Scholar
Cited by (1)
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse