method
active
method:attention-sink-scoreAttention Sink Score
Per-head metric measuring fraction of attention weight concentrated on first token position
Neighborhood — ranked by edge-count
Papers (1)
paper
Methods (1)
method
- Attention sink analysisrelated_toObserving tokens with abnormally high average activation to locate where R1 treats the response as starting.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Phenomenon where layers concentrate most attention weight on the first token, captured by ColSum Concentration metric
- Counting heads default attention to previous newline.
- Term introduced by this paper for the horizontal axis of information flow in transformers across token-positions via attention heads
- Fraction of attention heads with sink score above threshold 0.3, used to track stages of inference
- Core insight.
- Attention sinks in R1 appear toward the end of the response prefix rather than at the start of responsesfinding0.745Suggests R1 does not model its true response as begun until after the prefix.
- Decomposition approach with rank-1 OV directions.
- Process using Q, K, V to compute a heat map over K and weighted sum of V.