method
active
method:attention-sink-analysisAttention sink analysis
Observing tokens with abnormally high average activation to locate where R1 treats the response as starting.
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Attention SinkaboutPhenomenon where layers concentrate most attention weight on the first token, captured by ColSum Concentration metric
Methods (1)
method
- Attention Sink Scorerelated_toPer-head metric measuring fraction of attention weight concentrated on first token position
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Counting heads default attention to previous newline.
- Process using Q, K, V to compute a heat map over K and weighted sum of V.
- Term introduced by this paper for the horizontal axis of information flow in transformers across token-positions via attention heads
- Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
- Analysis measuring whether each attention head's maximum attention increase points to the correct injected sentence
- Attention patterns scaled by the norm of the value vector at each source position, showing how large a vector is moved from each position
- Decrease in attention paid to system prompt over conversational turns, leading to persona fidelity degradation (cited from Li et al.)
- Decomposition approach with rank-1 OV directions.