method
active
method:sparse-autoencoders-saeSparse Autoencoders (SAE)
Interpretability method criticized in this paper for shattering manifolds into atomic pieces, obscuring overarching semantic structure.
Neighborhood — ranked by edge-count
Papers (2)
paper
- Interpreting Language Model Parameterscitesmentions
- The World Inside Neural Networksmentions
Frameworks (2)
framework
- Natural Language Autoencoders (NLA)associated_withAn unsupervised method for generating natural language explanations of LLM activations through a verbalizer-reconstructor pair trained jointly with RL.
- The main framework proposed for retrieving and steering high-order semantic features in LLMs via sparse autoencoders.
Concepts (1)
concept
- Linear representationimplementsThe idea that features are encoded as directions in activation space.
Methods (4)
method
- TopK Sparse Autoencoders (SAEs)related_toSparse dictionary learning method used to extract interpretable features from EEG transformer embeddings.
- Adversarial Parameter Decomposition (VPD)associated_withCore technique introduced in this paper for decomposing neural network weight matrices into mechanistically simple, interpretable rank-one subcomponents.
- Natural Language Autoencoders (NLAs)associated_withCore unsupervised method for generating natural language explanations of LLM activations through a verbalizer-reconstructor pair trained with RL.
- Specific application of SAE to extract features from the middle layer of Claude 3 Sonnet, at three scales (1M, 4M, 34M features).
Claims (3)
claim
- Core critique of sparse autoencoders: they break the geometric structure of representations, making it harder to see the big picture.
- Sparse autoencoders don't provide a comprehensive solution because they decode activations, not parametersassociated_withCritique of activation-based interpretability methods.
- Core proposition of the paper: a substrate-level critique of existing interpretability methods.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Standard interpretability approach that VPD critiques and proposes an alternative to.
- Interpretability framework used to decompose layer-40 activations into sparse feature sets for studying emotional alignment and persistence
- The primary novel framework introduced in the paper for learning facet-level personality control vectors
- Used in Anthropic welfare assessment to identify performative behavior and hidden emotional struggle co-activating features
- Central claim of the paper: the method scales to state-of-the-art transformers.
- Primary method introduced: trains a one-hidden-layer MLP with L1 sparsity penalty to decompose model activations into overcomplete feature dictionaries
- Sparse Autoencoders Find Highly Interpretable Features in Language Models (Cunningham et al., 2023)concept0.825Core methodology paper for SAE-based interpretable feature extraction
- The paper's primary mechanistic analysis method: comparing SAE latent activations before and after fine-tuning to identify misalignment-relevant features