hypothesis
active
hypothesis:some-attentional-features-are-spread-across-multiple-headsSome attentional features are spread across multiple heads
Suspected but no confirmed example in real LLM.
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Attentional features spread across multiple heads.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Claim supported by VPD's recovery of cross-head attention subcomponents, noted in footnote.
- Expectation for decomposition.
- Proposed attention analog of sparse monosemantic features for decomposing attention layers.
- Empirical observation from examining expanded OV/QK matrices; approximately 10 out of 12 heads show significant copying
- Attention computations distribute across heads via parameter subcomponents with interpretable rolesfinding0.814Mechanistic discovery about how attention mechanisms decompose into interpretable parameter components.
- Interesting special case of copying behavior related to tokenization artifacts; primitive precursor to induction heads
- Are heads mediating the same edge attending for the same reason, suggesting attention head superposition?question0.805Distributed functional units.
- Long-standing bottleneck in mechanistic interpretability that VPD addresses by working natively on attention weight matrices.