concept
active
concept:attention-sinkAttention Sink
Phenomenon where layers concentrate most attention weight on the first token, captured by ColSum Concentration metric
Neighborhood — ranked by edge-count
Papers (2)
paper
Methods (1)
method
- Observing tokens with abnormally high average activation to locate where R1 treats the response as starting.
Concepts (2)
concept
- Newline attention sinkrelated_toCounting heads default attention to previous newline.
- Common opening of R1 reasoning traces that the model treats as part of the prompt.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Per-head metric measuring fraction of attention weight concentrated on first token position
- Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
- Term introduced by this paper for the horizontal axis of information flow in transformers across token-positions via attention heads
- Process using Q, K, V to compute a heat map over K and weighted sum of V.
- Decrease in attention paid to system prompt over conversational turns, leading to persona fidelity degradation (cited from Li et al.)
- Core operation in transformers, computing weighted combinations of previous elements
- Blocking attention edges to trace information flow.
- Attention sinks in R1 appear toward the end of the response prefix rather than at the start of responsesfinding0.752Suggests R1 does not model its true response as begun until after the prefix.