thinker
active
thinker:yoav-gur-arieh

Yoav Gur-Arieh

Authored
1
Introduces
0
Studies
0
Affiliations
2
Cited by
0

Authored papers (1)

  • Language models retrieve bound entities through a mixture of three mechanisms, not the positional mechanism alone, and the positional signal collapses in the middle of long entity lists. With n=20 entity groups on gemma-2-2b-it, positional patching controls behavior only at the first and last groups and explains roughly 20% of behavior in middle groups, where its logit distribution turns wide and diffuse. A lexical mechanism (query entity retrieves its bound counterpart) and a reflexive mechanism (a direct self-pointer to the answer token) supply the sharp, one-hot corrections. Which one dominates depends on where the target sits in its clause: reflexive for t_entity=1, lexical for t_entity=3. The paper introduces a counterfactual interchange-intervention design (TargetRebind) that makes each mechanism predict a different token. It also introduces a causal model that mixes a Gaussian positional term, whose variance is quadratic in position, with one-hot lexical and reflexive terms. That model reaches 0.95 mean Jensen–Shannon similarity with the LM's next-token distribution, against 0.44 for the one-hot positional view, which falls below the 0.5 uniform baseline. The pattern holds across nine models (Gemma-2, Qwen2.5, Llama-3.1; 2–72B) and ten tasks, and it persists when up to 10,000 filler tokens are interleaved. Binding retrieval is therefore a position-weighted ensemble. Weakening lexical signal against a noisier positional one is offered as a mechanistic account of lost-in-the-middle.

More papers — OpenAlex / S2

Affiliations (2)

Co-authors (2)

Recent mentions (1)