hypothesis
active
hypothesis:special-tokens-have-very-large-magnitude-so-their-sae-features-also-have-large-magnitude-and-sit-far-from-other-pointsSpecial tokens have very large magnitude, so their SAE features also have large magnitude and sit far from other points
First explanation for isolated features.
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (1)
finding
- Observation on the Llama 3.3 70B SAE feature map.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Motivation.
- Core critique of sparse autoencoders: they break the geometric structure of representations, making it harder to see the big picture.
- The specific SAE architecture trained: 100K+ possible features compressed to 64 active per token for layer-40 activations
- Scaling SAE size increases granularity and discovers new features.
- Unsure.
- Core concern.
- Basic SAE performance metrics.
- Chat-trained SAEs find behaviourally relevant features because many causal features occur on chat-specific special tokenshypothesis0.765Explanation for LMSYS superiority.