finding
active
finding:attention-sinks-in-r1-appear-toward-the-end-of-the-response-prefix-rather-than-at-the-start-of-responsesAttention sinks in R1 appear toward the end of the response prefix rather than at the start of responses
Suggests R1 does not model its true response as begun until after the prefix.
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Interpretation of the prefix and attention sink results.
Hypotheses (1)
hypothesis
- Explanation for late steering and attention sinks.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Evidence for the training-prefix hypothesis.
- Supports the view that R1's boundaries differ from user-perceived ones.
- R1 implicitly recognizes confusion or incoherence when activations are overly perturbed, causing it to stop and course-correcthypothesis0.757Working hypothesis for the oversteering reversion.
- Phenomenon where layers concentrate most attention weight on the first token, captured by ColSum Concentration metric
- First steering insight specific to R1.
- Architectural explanation.
- Per-head metric measuring fraction of attention weight concentrated on first token position
- Oversteering some R1 features makes the model revert to original behavior before outputs become incoherentfinding0.744Second steering insight, unlike non-reasoning models.