Previous Token Head

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• A pair of query and key subcomponents distributed across attention heads performs previous-token behaviorfinding0.774
  VPD recovers an attention algorithm for attending to the previous token, distributed across multiple heads.
• Previous-token attention behaviorconcept0.770
  An attention algorithm recovered by VPD where the model attends to the immediately preceding token.
• Tokenconcept0.748
  Basic unit of LLM input/output: words, parts of words, punctuation marks, emojis
• Next Token Predictionconcept0.747
  The training objective of LLMs: predicting the most likely next token given context; formally P(w_{n+1}|w_1...w_n)
• Induction heads work by using K-composition with a previous token head to shift keys by one token, then matching the current destination token against shifted keys to predict what followsclaim0.720
  The mechanistic explanation of how induction heads are implemented in two-layer models
• Token-in-Context Featureconcept0.712
  Feature that fires on a specific token only within a specific surrounding context (e.g., 'the' in physics vs 'the' in mathematics)
• All-token steeringmethod0.700
  Baseline steering method that applies intervention at every token generation step, shown to degrade performance at high strengths
• Token embeddingsconcept0.698
  Vector representations of individual tokens from genomic foundation models; the raw inputs to sequence pooling methods.