claim
active
claim:multiple-attention-heads-are-required-to-construct-the-curved-counting-manifold-because-a-single-head-is-constrained-by-input-curvatureMultiple attention heads are required to construct the curved counting manifold because a single head is constrained by input curvature
Distributed algorithm.
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (3)
finding
- Distributed counting.
- Justifies focusing on attention.
- Sum of near-1D head outputs.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Empirical observation from examining expanded OV/QK matrices; approximately 10 out of 12 heads show significant copying
- Interesting special case of copying behavior related to tokenization artifacts; primitive precursor to induction heads
- Theoretical point.
- Proposes that nonlinear geometric structure is superior to linear feature spaces for capturing semantic content.
- Key decomposition enabling separate analysis of where attention goes and what it does
- Are heads mediating the same edge attending for the same reason, suggesting attention head superposition?question0.748Distributed functional units.
- Quantitative result from eigenvalue analysis of expanded OV matrices; confirmed by qualitative inspection
- Multiple boundary heads have distinct overlapping response curves that tile characters remainingfinding0.747Stereoscopic algorithm.