finding
active
finding:increasing-topk-sparsity-drives-synthetic-saes-through-shattering-capture-then-dilution-regimesIncreasing TopK sparsity drives synthetic SAEs through shattering, capture, then dilution regimes
Phase diagram on support size and receptive field spread.
Source paper
extracted_from(2026) · Usha Bhalla · Thomas Fel · Can Rager · Sheridan Feucht +8
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- SAEs can capture manifolds either globally via a compact atom group spanning the manifold or locally via features tiling regionsassociated_withsupportsTheoretical framework answer.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Sparse dictionary learning method used to extract interpretable features from EEG transformer embeddings.
- Evidence in real models.
- Current SAEs mix global subspace and local tiling solutions in a fragmented dilution regimeclaim0.784Central empirical interpretation.
- SAEs assume sparsity (number of concepts) is time-invariant, drawn i.i.d. from a fixed distribution.claim0.751Corollary 4.1.1.
- Central result.
- Supervised metrics discriminate.
- Subspace capture on synthetic manifolds peaks near k=4 and degrades at lower and higher sparsityfinding0.745Sparsity sweet spot in synthetic benchmark (R^128, c=512).
- Core critique of sparse autoencoders: they break the geometric structure of representations, making it harder to see the big picture.