We reproduce the results in Meng et al. (2022)'s Figure 1 of locating early sites and late sites of factual associations in GPT2-XL in about 20 lines of pyvene code.

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• pyvene reproduces Meng et al. 2022 Figure 1 (factual association localization in GPT2-XL) in about 20 lines of codefinding0.924
  Case Study I demonstrating pyvene can replicate a major interpretability result compactly
• Locating and Editing Factual Associations in GPT (Meng et al., 2022)concept0.824
  Cited as causal intervention methodology precedent
• Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small (Wang et al., 2023)concept0.739
  Cited as causal intervention methodology precedent for this paper's ablation approach
• GPT-2 implements at least one induction head using pointer arithmetic on positional embeddings rather than K-compositionhypothesis0.737
  Observation of an alternative induction head implementation algorithm in larger models with positional embeddings in the residual stream
• Positional encodings inferred on the fly from previously learned structures would offer fruitful research direction for language, maths, and logicclaim0.730
  Forward-looking interpretive claim about the implications of recurrent position encodings for NLP research.
• GPT does not generate rollouts during training, so there is no reason to expect that GPT will form preferences over the consequences of its output related to the text prediction objective.claim0.728
  Argues against instrumental convergence in GPT.
• GPT's ability to simulate text automata is the source of its most surprising and pivotal implications for paths to superintelligence.claim0.722
  Importance of recursive generation.
• Can GPT distinguish correlation and causality?question0.718
  Disambiguation exercise.