Induction Heads

Mechanistic circuits in transformers documented by Olsson et al. 2022, cited as evidence for pattern-repository assumption

Neighborhood — ranked by edge-count

paper

thinker

Chris Olah
introduces
Co-author; provided high-level research guidance, wrote introduction/discussion.

concept

in-context learning (ICL)
associated_withimplements
Test-time adaptation from prompt or retrieved context with no parameter updates.
Previous Token Head
associated_withimplements
An attention head that primarily attends to the immediately preceding token; key building block for induction heads via K-composition
Skip-Trigram
extends
A three-token pattern of the form [source]...[destination][out] that one-layer attention heads implement; the paper's key characterization of one-layer transformer behavior
K-Composition
implements
A form of attention head composition where W_K reads from a subspace affected by a previous head; central to how induction heads are implemented
latent pattern repository (Pprior)
supports
Unlabeled statistical regularities stored during pretraining.
Two-Layer Attention-Only Transformer
implements
The primary model analyzed; uses attention head composition, especially K-composition, to create induction heads for powerful in-context learning

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

In-Context Learning and Induction Heads (forthcoming paper)concept0.772
A follow-up paper extending the framework and induction head concept to larger more realistic models
Induction heads explain in-context learning in small models and only develop in models with at least two attention layersclaim0.749
Central empirical claim of the paper; induction heads are shown to be the mechanism for powerful in-context learning
Induction heads work by using K-composition with a previous token head to shift keys by one token, then matching the current destination token against shifted keys to predict what followsclaim0.734
The mechanistic explanation of how induction heads are implemented in two-layer models
Large models form many induction heads built from K-composition with a previous token head, making induction heads a central driver of in-context learning at all scalesclaim0.732
Forward-looking claim connecting toy model findings to large-scale language models
Deep Model Inductionframework0.730
Natural Inductionconcept0.726
Emerging framework that seeks invariants between evolution and learning; cited as future direction.
Attention headsconcept0.724
Transformer attention heads that could be recruited to extract different kinds of information (text vs. thoughts).
Concordance headsconcept0.714
QK circuit heads hypothesized to measure likelihood of an output given prior activations, used in prefill detection.