paper:doi-10-48550-arxiv-2409-14507A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
Original abstract (expand)
Sparse Autoencoders (SAEs) aim to decompose the activation space of large language models (LLMs) into human-interpretable latent directions or features. As we increase the number of features in the SAE, hierarchical features tend to split into finer features ("math"may split into"algebra","geometry", etc.), a phenomenon referred to as feature splitting. However, we show that sparse decomposition and splitting of hierarchical features is not robust. Specifically, we show that seemingly monosemantic features fail to fire where they should, and instead get"absorbed"into their children features. We coin this phenomenon feature absorption, and show that it is caused by optimizing for sparsity in SAEs whenever the underlying features form a hierarchy. We introduce a metric to detect absorption in SAEs, and validate our findings empirically on hundreds of LLM SAEs. Our investigation suggests that varying SAE sizes or sparsity is insufficient to solve this issue. We discuss the implications of feature absorption in SAEs and some potential approaches to solve the fundamental theoretical issues before SAEs can be used for interpreting LLMs robustly and at scale.
Similar preprints — Semantic Scholar
Cited by (5)
- Stochastic Parameter Decomposition
Rank-one subcomponents trained against stochastic ablation masks recover ground-truth mechanisms in toy networks that Attribution-based Parameter Decomposition (APD) could not decompose, while removin
- Understanding sparse autoencoder scaling in the presence of feature manifolds
Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead
- Do Sparse Autoencoders Capture Concept Manifolds?
Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept
- When Models Manipulate Manifolds: The Geometry of a Counting Task
Claude 3.5 Haiku's linebreaking in fixed-width text is implemented by manipulating low-dimensional, rippled 1-D feature manifolds: character count, line width, characters remaining and next-token leng
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
Gains on the sparsity-fidelity frontier do not predict how well a sparse autoencoder performs on downstream tasks, and SAEBench demonstrates this by scoring over 200 SAEs on eight metrics. Matryoshka