framework
active
framework:sparse-autoencoder-for-dictionary-learningSparse Autoencoder for Dictionary Learning
Primary method introduced: trains a one-hidden-layer MLP with L1 sparsity penalty to decompose model activations into overcomplete feature dictionaries
Neighborhood — ranked by edge-count
Papers (1)
paper
- Toy Models of Superpositionextends
Thinkers (3)
thinker
- Lee Sharkeystudies
- Trenton Brickenstudies
- Hoagy CunninghamstudiesCo-author; de-risked residual-stream SAEs, ran feature completeness analysis.
Methods (1)
method
- Sparse Dictionary LearningimplementsGeneral method for finding overcomplete sparse decompositions; the paper uses sparse autoencoders as an approximation
Concepts (2)
concept
- Pre-Encoder Biasassociated_withArchitectural modification subtracting a learned bias from autoencoder inputs before encoding; initialized to geometric median of dataset; improves autoencoder performance
- Untied Decoder Weightsassociated_withAutoencoder design choice to learn separate encoder and decoder weights, increasing representational capacity by allowing encoder vectors to distinguish similar features
Claims (1)
claim
- Rationale for using simpler sparse autoencoders rather than NP-hard compressed sensing algorithms
Frameworks (3)
framework
- Sparse Autoencoderrelated_toInterpretability framework used to decompose layer-40 activations into sparse feature sets for studying emotional alignment and persistence
- Superposition HypothesisimplementsCore theoretical framework: neural networks represent more features than neurons by encoding features as directions in superposition
- Compressed SensingextendsMathematical framework enabling recovery of high-dimensional sparse vectors from low-dimensional projections; theoretically underpins sparse autoencoder approach
Artifacts (1)
artifact
- Interactive interface for exploring all 90 learned dictionaries' features, including activating examples, logit effects, and ablations
Conceptual bridges
2-hop · via this framework's ideasWhere ideas in this framework connect to the rest of the corpus — the same concept, an analogy, or a restatement elsewhere.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Candidate approach to solving superposition.
- Used in Anthropic welfare assessment to identify performative behavior and hidden emotional struggle co-activating features
- Interpretability method criticized in this paper for shattering manifolds into atomic pieces, obscuring overarching semantic structure.
- Empirical principle discovered during autoencoder training; led to using 8 billion training points
- The central mechanistic interpretability tool applied across all three EEG transformers to extract sparse feature dictionaries
- Extension of SAEs that jointly learns latents across representations from different models; proposed as alternative for extended fine-tuning
- Sparse dictionary learning variant that reads and writes to multiple layers or models, yielding shared features.
- Interpretability tool underlying top-feature, details, steering tools.