question
active
question:are-heads-mediating-the-same-edge-attending-for-the-same-reason-suggesting-attention-head-superpositionAre heads mediating the same edge attending for the same reason, suggesting attention head superposition?
Distributed functional units.
Source paper
extracted_fromNeighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Possibility that functional units are distributed across heads.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Suspected but no confirmed example in real LLM.
- Theoretical point.
- Motivation.
- Claim supported by VPD's recovery of cross-head attention subcomponents, noted in footnote.
- Extension of superposition hypothesis to attention layers as future research direction
- Will a head perform 'the same algorithm' across prompts, or are individual heads polysemantic?question0.777Head polysemanticity.
- Motivates including Head Cor+Anti condition in steering position comparison
- Empirical observation from examining expanded OV/QK matrices; approximately 10 out of 12 heads show significant copying