finding
active
finding:previous-token-heads-are-driven-by-query-side-first-name-features-interacting-with-key-side-title-preceding-name-featuresPrevious-token heads are driven by query-side 'first name' features interacting with key-side 'title preceding name' features
Precursor to induction.
Source paper
extracted_fromNeighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Induction HeadssupportsMechanistic circuits in transformers documented by Olsson et al. 2022, cited as evidence for pattern-repository assumption
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- An attention head that primarily attends to the immediately preceding token; key building block for induction heads via K-composition
- A pair of query and key subcomponents distributed across attention heads performs previous-token behaviorfinding0.807VPD recovers an attention algorithm for attending to the previous token, distributed across multiple heads.
- The mechanistic explanation of how induction heads are implemented in two-layer models
- An attention algorithm recovered by VPD where the model attends to the immediately preceding token.
- Interesting special case of copying behavior related to tokenization artifacts; primitive precursor to induction heads
- QK attribution of induction-relevant heads.
- Forward-looking claim connecting toy model findings to large-scale language models
- QK attribution for MC heads.