thinker
active
thinker:mor-geva

Mor Geva

Authored
2
Introduces
0
Studies
0
Affiliations
1
Cited by
6

Authored papers (2)

  • Language models retrieve bound entities through a mixture of three mechanisms, not the positional mechanism alone, and the positional signal collapses in the middle of long entity lists. With n=20 entity groups on gemma-2-2b-it, positional patching controls behavior only at the first and last groups and explains roughly 20% of behavior in middle groups, where its logit distribution turns wide and diffuse. A lexical mechanism (query entity retrieves its bound counterpart) and a reflexive mechanism (a direct self-pointer to the answer token) supply the sharp, one-hot corrections. Which one dominates depends on where the target sits in its clause: reflexive for t_entity=1, lexical for t_entity=3. The paper introduces a counterfactual interchange-intervention design (TargetRebind) that makes each mechanism predict a different token. It also introduces a causal model that mixes a Gaussian positional term, whose variance is quadratic in position, with one-hot lexical and reflexive terms. That model reaches 0.95 mean Jensen–Shannon similarity with the LM's next-token distribution, against 0.44 for the one-hot positional view, which falls below the 0.5 uniform baseline. The pattern holds across nine models (Gemma-2, Qwen2.5, Llama-3.1; 2–72B) and ten tasks, and it persists when up to 10,000 filler tokens are interleaved. Binding retrieval is therefore a position-weighted ensemble. Weakening lexical signal against a noisier positional one is offered as a mechanistic account of lost-in-the-middle.

  • Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.

More papers — OpenAlex / S2

Affiliations (1)

Recent mentions (1)