The mathematical framework and induction head concept will remain at least partially relevant for larger, more realistic models

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Large models form many induction heads built from K-composition with a previous token head, making induction heads a central driver of in-context learning at all scalesclaim0.777
  Forward-looking claim connecting toy model findings to large-scale language models
• The earlier a base model (less exposure to LM-related data), the more it is surprised by its own spontaneous self-referential capabilities.claim0.755
  Claim that capability emerges from architecture, not data, and that later models lose the surprise.
• If a specific framing, which uses tools normally reserved for brains, results in fruitful new research programs on unconventional systems, then the scientific approach requires that we consider those systems to be bona fide subjects of that corner of the natural world.hypothesis0.755
  Methodological hypothesis from Box 1: the pragmatic test for extending cognitive terminology.
• Conceptual advances in the links between machine learning and evolution now provide quantitative formalisms with which to begin to develop testable models of collective intelligence across scales.claim0.754
  Asserts that the time is ripe for formal models.
• All induction heads fall in an extreme corner of high OV eigenvalue positivity and high QK eigenvalue positivity, confirming the mechanistic theoryclaim0.747
  Quantitative verification that the copying and matching structure predicted by the mechanistic theory is present in all observed induction heads
• What if the concept being manipulated does not lie on a straight line in the model's representations?question0.746
  The motivating question that opens the paper and leads to the development of manifold steering.
• Any system that persists must minimise surprisal, thereby gathering evidence for its own generative model.quote0.745
  Opening sentence defining self-evidencing.
• Features can be used to steer large models.claim0.744
  Clamping feature activations causally alters model behavior in interpretable ways.