Attention Sink

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Attention Sink Scoremethod0.885
  Per-head metric measuring fraction of attention weight concentrated on first token position
• Attentionconcept0.788
  Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
• Attention streamsconcept0.784
  Term introduced by this paper for the horizontal axis of information flow in transformers across token-positions via attention heads
• attention computationconcept0.767
  Process using Q, K, V to compute a heat map over K and weighted sum of V.
• Attention Decayconcept0.764
  Decrease in attention paid to system prompt over conversational turns, leading to persona fidelity degradation (cited from Li et al.)
• attention mechanismconcept0.754
  Core operation in transformers, computing weighted combinations of previous elements
• Attention headsconcept0.746
  Transformer attention heads that could be recruited to extract different kinds of information (text vs. thoughts).
• paying attention to the wholenessconcept0.743
  The act of seeing and feeling the entire field of centers at a place, which Alexander equates with love of life.