finding
active
finding:approximately-30-of-llama-3-1-8b-sae-features-and-3-5-of-llama-3-3-70b-sae-features-were-removed-including-dead-featuresApproximately 30% of Llama 3.1 8B SAE features and 3.5% of Llama 3.3 70B SAE features were removed, including dead features
Moderation statistics.
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Harmful features mostly reflect graphic criminal content subsets of the LMSys chat training datasupportsExplanation for moderated features.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Goodfire blog post describing SAEs used for Llama models in this study
- C.5.1.
- Most features dead in largest SAE, indicating room for improvement.
- One of four LLMs selected; larger model with D=8192 embedding dimension; analyzed across proportionally aligned layers.
- One of four LLMs selected for representation analysis; embedding dimension D=4096; used as demonstration model in scatter plots.
- The model used in Experiment 2 for SAE feature steering experiments via Goodfire API
- Seen as in other SAEs.
- Model-specific difference in persona susceptibility