claim
active
claim:moe-models-should-have-less-superposition-because-they-eat-the-gap-between-neuron-and-feature-sparsityMoE models should have less superposition because they eat the gap between neuron and feature sparsity
Strategic Approach 1.
Neighborhood — ranked by edge-count
Papers (1)
paper
- Toy Models of Superpositionmentions
Claims (1)
claim
- Models significantly benefit from superposition, effectively making models much biggerassociated_withCost of removing superposition.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Major prediction checked against InceptionV1.
- Interpretation of the cars-in-superposition circuit finding as an intentional representational strategy
- Counter-example disproving that architectural sparsity alone can prevent polysemanticity
- Superposition hypothesis: neural networks represent more features than dimensions using almost-orthogonal directions.hypothesis0.793Explanation for why dictionary learning can recover many more features than dimensions.
- Figure 4.
- Superposition hypothesis: networks represent more features than neurons as almost-orthogonal directionshypothesis0.783Central hypothesis.
- Superposition controls robustness.
- Core demonstration.