concept
active
concept:induction-headsInduction Heads
Mechanistic circuits in transformers documented by Olsson et al. 2022, cited as evidence for pattern-repository assumption
Neighborhood — ranked by edge-count
Papers (3)
paper
Thinkers (1)
thinker
- Chris Olahintroduces
Concepts (8)
concept
- in-context learning (ICL)associated_withimplementsTest-time adaptation from prompt or retrieved context with no parameter updates.
- Previous Token Headassociated_withimplementsAn attention head that primarily attends to the immediately preceding token; key building block for induction heads via K-composition
- K-CompositionimplementsA form of attention head composition where W_K reads from a subspace affected by a previous head; central to how induction heads are implemented
- Skip-TrigramextendsA three-token pattern of the form [source]...[destination][out] that one-layer attention heads implement; the paper's key characterization of one-layer transformer behavior
- Unlabeled statistical regularities stored during pretraining.
- QK Circuitassociated_withThe bilinear circuit formed by W_E^T W_QK^h W_E that determines which source token a destination token attends to
- Two-Layer Attention-Only TransformerimplementsThe primary model analyzed; uses attention head composition, especially K-composition, to create induction heads for powerful in-context learning
- Fuzzy inductionextendsInduction performed in the feature basis rather than token space.
Findings (1)
finding
- Precursor to induction.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Fuzzy induction heads match words across languages, antonyms, structural positions and kinship termsfinding0.796Variety of fuzzy matching found.
- A follow-up paper extending the framework and induction head concept to larger more realistic models
- Central empirical claim of the paper; induction heads are shown to be the mechanism for powerful in-context learning
- Hypothesized heads that invert semantic meaning of features; not confirmed.
- Attention heads whose QK twists count manifolds to detect approaching line end.
- The mechanistic explanation of how induction heads are implemented in two-layer models
- Forward-looking claim connecting toy model findings to large-scale language models