concept
active
concept:talking-heads-attention

Talking Heads Attention

A transformer variant where OV and QK matrices of different attention heads can share components, enabling shared copying mechanisms

Neighborhood — ranked by edge-count

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

  • Attention headsconcept0.838
    Transformer attention heads that could be recruited to extract different kinds of information (text vs. thoughts).
  • Possibility that functional units are distributed across heads.
  • The composition of two attention heads via V-composition, forming a new entity with its own attention pattern A^h2 * A^h1 and OV matrix W_OV^h2 * W_OV^h1
  • Analysis measuring whether each attention head's maximum attention increase points to the correct injected sentence
  • Attentionconcept0.731
    Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
  • Self-attentionconcept0.719
    A form of key-query attention within a single input sequence; core to Transformers.
  • Induction Headsconcept0.709
    Mechanistic circuits in transformers documented by Olsson et al. 2022, cited as evidence for pattern-repository assumption
  • Suspected but no confirmed example in real LLM.