method
active
method:attention-sink-analysis

Attention sink analysis

Observing tokens with abnormally high average activation to locate where R1 treats the response as starting.

Neighborhood — ranked by edge-count

Concepts (1)

concept
  • Phenomenon where layers concentrate most attention weight on the first token, captured by ColSum Concentration metric

Methods (1)

method
  • Per-head metric measuring fraction of attention weight concentrated on first token position

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

  • Counting heads default attention to previous newline.
  • Process using Q, K, V to compute a heat map over K and weighted sum of V.
  • Attention streamsconcept0.761
    Term introduced by this paper for the horizontal axis of information flow in transformers across token-positions via attention heads
  • Attentionconcept0.746
    Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
  • Analysis measuring whether each attention head's maximum attention increase points to the correct injected sentence
  • Attention patterns scaled by the norm of the value vector at each source position, showing how large a vector is moved from each position
  • Attention Decayconcept0.741
    Decrease in attention paid to system prompt over conversational turns, leading to persona fidelity degradation (cited from Li et al.)
  • Attention-output SAEsframework0.736
    Decomposition approach with rank-1 OV directions.