concept
active
concept:virtual-attention-headVirtual Attention Head
The composition of two attention heads via V-composition, forming a new entity with its own attention pattern A^h2 * A^h1 and OV matrix W_OV^h2 * W_OV^h1
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Mathematical equivalence showing the relationship between attention mechanisms and convolutional operations
Concepts (1)
concept
- V-CompositionimplementsA form of attention head composition where W_V reads from a subspace affected by a previous head, creating virtual attention heads
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Transformer attention heads that could be recruited to extract different kinds of information (text vs. thoughts).
- Analysis measuring whether each attention head's maximum attention increase points to the correct injected sentence
- Possibility that functional units are distributed across heads.
- A transformer variant where OV and QK matrices of different attention heads can share components, enabling shared copying mechanisms
- Suspected but no confirmed example in real LLM.
- Expectation for decomposition.
- Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
- A form of key-query attention within a single input sequence; core to Transformers.