concept
active
concept:newline-attention-sinkNewline attention sink
Counting heads default attention to previous newline.
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Attention Sinkrelated_toPhenomenon where layers concentrate most attention weight on the first token, captured by ColSum Concentration metric
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Observing tokens with abnormally high average activation to locate where R1 treats the response as starting.
- Per-head metric measuring fraction of attention weight concentrated on first token position
- Term introduced by this paper for the horizontal axis of information flow in transformers across token-positions via attention heads
- Circuit interpretation.
- Classifiers used for token filtering and reward shaping.
- Blocking attention edges to trace information flow.
- Process using Q, K, V to compute a heat map over K and weighted sum of V.
- Core operation in transformers, computing weighted combinations of previous elements