Key-query attention

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Grouped Query Attentionframework0.805
  Variant of multi-head attention used in Llama3 and Qwen2.5 models evaluated in this paper
• Attentionconcept0.769
  Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
• Attention Schemaconcept0.745
  A predictive model representing and controlling attention; central to attention schema theory.
• Attention Sinkconcept0.741
  Phenomenon where layers concentrate most attention weight on the first token, captured by ColSum Concentration metric
• attention computationconcept0.739
  Process using Q, K, V to compute a heat map over K and weighted sum of V.
• answer.querymethod0.733
  An Elephant action of answering a query.
• Position-Only Keys/Queries, Stimulus-Only Values Factorizationmethod0.732
  Key architectural modification restricting queries and keys to position encodings while values depend only on stimuli; extreme version of best-practice insight.
• A pair of query and key subcomponents distributed across attention heads performs previous-token behaviorfinding0.727
  VPD recovers an attention algorithm for attending to the previous token, distributed across multiple heads.