claim
active
claim:a-sum-of-rank-1-heads-cannot-approximate-every-real-head-wellA sum of rank-1 heads cannot approximate every real head well
Implies features of rank greater than one.
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Features whose QK/OV are high rank, hard to interpret.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Theme across prior and own work.
- Will a head perform 'the same algorithm' across prompts, or are individual heads polysemantic?question0.760Head polysemanticity.
- Central theme.
- Interpretation of steering asymmetry.
- MC interpretation.
- Empirical observation from examining expanded OV/QK matrices; approximately 10 out of 12 heads show significant copying
- A random head in the same layer shows almost no structure between count and width probesfinding0.732Control for twisting.
- Head-level causal evidence.