claim
active
claim:failure-to-reach-sparsity-reflects-the-basis-not-the-potential-for-sparse-interpretation-no-saliency-scheme-will-do-much-betterFailure to reach sparsity reflects the basis, not the potential for sparse interpretation; no saliency scheme will do much better
Better bases may give sparsity.
Source paper
extracted_fromNicholas L. Turner · Jeffrey Wu · Joshua Batson
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (3)
finding
- Sampled helpfulness mass.
- Parameter count roughly matches original transformer.
- Estimates density floor.
Hypotheses (1)
hypothesis
- Same model can be dense in one basis and sparse in another.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Main proposal.
- Playful and poetic amplification support.
- SDL assumption may break down.
- Discovery story.
- Motivates multi-metric evaluation.
- Major prediction checked against InceptionV1.
- Higher sparsity forces composition of multiple concepts into fewer active latents, harming disentanglementhypothesis0.781Explains TPP/RAVEL preferring high L0.
- Lower SAE loss at fixed sparsity likely indicates less interference and less superposition.claim0.779Justifies SAE proxy.