concept
active
concept:previous-token-attention-behaviorPrevious-token attention behavior
An attention algorithm recovered by VPD where the model attends to the immediately preceding token.
Neighborhood — ranked by edge-count
Papers (1)
paper
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- A pair of query and key subcomponents distributed across attention heads performs previous-token behaviorfinding0.842VPD recovers an attention algorithm for attending to the previous token, distributed across multiple heads.
- An attention head that primarily attends to the immediately preceding token; key building block for induction heads via K-composition
- Precursor to induction.
- Blocking attention to query token in layers 11–16 drops accuracy from 98% to 37%; with next token 8%finding0.748Attention knockout.
- Behavior where information about full clauses is encoded over clause-ending punctuation tokens in LLMs
- The training objective of LLMs: predicting the most likely next token given context; formally P(w_{n+1}|w_1...w_n)
- Reward probes attend to entity and relation tokens; Failed Fix shows higher attention on punctuationfinding0.734Attention map observation.
- Token-level supervision enables models to learn functional-token invocation from reasoning contextclaim0.730ATLAS author's assertion that functional tokens optimized via standard cross-entropy loss learn when and how to invoke operations from surrounding text.