GPT-2 implements at least one induction head using pointer arithmetic on positional embeddings rather than K-composition

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Induction heads work by using K-composition with a previous token head to shift keys by one token, then matching the current destination token against shifted keys to predict what followsclaim0.807
  The mechanistic explanation of how induction heads are implemented in two-layer models
• Large models form many induction heads built from K-composition with a previous token head, making induction heads a central driver of in-context learning at all scalesclaim0.781
  Forward-looking claim connecting toy model findings to large-scale language models
• GPT-2concept0.755
  Early large language model cited as an example of transformer-based LLMs
• Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small (Wang et al., 2023)concept0.751
  Cited as causal intervention methodology precedent for this paper's ablation approach
• pyvene reproduces Meng et al. 2022 Figure 1 (factual association localization in GPT2-XL) in about 20 lines of codefinding0.743
  Case Study I demonstrating pyvene can replicate a major interpretability result compactly
• Induction heads explain in-context learning in small models and only develop in models with at least two attention layersclaim0.741
  Central empirical claim of the paper; induction heads are shown to be the mechanism for powerful in-context learning
• We reproduce the results in Meng et al. (2022)'s Figure 1 of locating early sites and late sites of factual associations in GPT2-XL in about 20 lines of pyvene code.quote0.737
  Load-bearing demonstration of pyvene's conciseness for complex replication tasks
• All induction heads fall in an extreme corner of high OV eigenvalue positivity and high QK eigenvalue positivity, confirming the mechanistic theoryclaim0.731
  Quantitative verification that the copying and matching structure predicted by the mechanistic theory is present in all observed induction heads