finding
active
finding:early-layer-induction-heads-do-literal-copying-while-mid-layer-heads-do-fuzzier-inductionEarly-layer induction heads do literal copying while mid-layer heads do fuzzier induction
Layer-dependent induction on Haiku 3.5.
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Central interpretation of induction case study.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Central empirical claim of the paper; induction heads are shown to be the mechanism for powerful in-context learning
- Empirical observation from examining expanded OV/QK matrices; approximately 10 out of 12 heads show significant copying
- The mechanistic explanation of how induction heads are implemented in two-layer models
- Strong test of the induction head hypothesis using uniformly sampled random tokens repeated three times
- Striking mechanistic finding that injection creates universally detectable perturbation in residual stream immediately downstream
- Quantitative verification that the copying and matching structure predicted by the mechanistic theory is present in all observed induction heads
- Quantitative verification of the mechanistic theory; both circuits required for the induction algorithm show the predicted copying/matching structure
- Fuzzy poetry induction head OV circuit is identical to literal induction (newline value to 'say \n' output)finding0.760Copying behavior unchanged.