Large models form many induction heads built from K-composition with a previous token head, making induction heads a central driver of in-context learning at all scales

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Induction heads explain in-context learning in small models and only develop in models with at least two attention layersclaim0.858
  Central empirical claim of the paper; induction heads are shown to be the mechanism for powerful in-context learning
• Induction heads work by using K-composition with a previous token head to shift keys by one token, then matching the current destination token against shifted keys to predict what followsclaim0.852
  The mechanistic explanation of how induction heads are implemented in two-layer models
• Induction heads in two-layer models successfully perform in-context learning on completely random repeated token sequences far outside training distributionfinding0.833
  Strong test of the induction head hypothesis using uniformly sampled random tokens repeated three times
• In-Context Learning and Induction Heads (forthcoming paper)concept0.799
  A follow-up paper extending the framework and induction head concept to larger more realistic models
• GPT-2 implements at least one induction head using pointer arithmetic on positional embeddings rather than K-compositionhypothesis0.781
  Observation of an alternative induction head implementation algorithm in larger models with positional embeddings in the residual stream
• The mathematical framework and induction head concept will remain at least partially relevant for larger, more realistic modelshypothesis0.777
  Central motivating hypothesis for the forthcoming paper on in-context learning and induction heads
• The Primer architecture's depthwise convolution change would allow induction heads to form without requiring K-compositionhypothesis0.777
  Architectural interpretation of how Primer's design change relates to the paper's mechanistic theory of induction heads
• All induction heads fall in an extreme corner of high OV eigenvalue positivity and high QK eigenvalue positivity, confirming the mechanistic theoryclaim0.769
  Quantitative verification that the copying and matching structure predicted by the mechanistic theory is present in all observed induction heads