Previous-token attention behavior

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• A pair of query and key subcomponents distributed across attention heads performs previous-token behaviorfinding0.842
  VPD recovers an attention algorithm for attending to the previous token, distributed across multiple heads.
• Previous Token Headconcept0.770
  An attention head that primarily attends to the immediately preceding token; key building block for induction heads via K-composition
• Summarization Token Behaviorconcept0.748
  Behavior where information about full clauses is encoded over clause-ending punctuation tokens in LLMs
• Next Token Predictionconcept0.740
  The training objective of LLMs: predicting the most likely next token given context; formally P(w_{n+1}|w_1...w_n)
• Token-level supervision enables models to learn functional-token invocation from reasoning contextclaim0.730
  ATLAS author's assertion that functional tokens optimized via standard cross-entropy loss learn when and how to invoke operations from surrounding text.
• Each functional token is associated with an internalized visual operation, yet requires no visual supervision and remains a standard token in the tokenizer vocabulary.claim0.728
  Describes the properties of the functional token.
• Token-in-Context Featureconcept0.728
  Feature that fires on a specific token only within a specific surrounding context (e.g., 'the' in physics vs 'the' in mathematics)
• Some attention heads partially specialize in copying for words that split into two tokens without a space prefix, attending from fragmented token to complete tokenfinding0.727
  Interesting special case of copying behavior related to tokenization artifacts; primitive precursor to induction heads