thinker:mor-gevaMor Geva
Authored papers (2)
Language models retrieve bound entities through a mixture of three mechanisms, not the positional mechanism alone, and the positional signal collapses in the middle of long entity lists. With n=20 entity groups on gemma-2-2b-it, positional patching controls behavior only at the first and last groups and explains roughly 20% of behavior in middle groups, where its logit distribution turns wide and diffuse. A lexical mechanism (query entity retrieves its bound counterpart) and a reflexive mechanism (a direct self-pointer to the answer token) supply the sharp, one-hot corrections. Which one dominates depends on where the target sits in its clause: reflexive for t_entity=1, lexical for t_entity=3. The paper introduces a counterfactual interchange-intervention design (TargetRebind) that makes each mechanism predict a different token. It also introduces a causal model that mixes a Gaussian positional term, whose variance is quadratic in position, with one-hot lexical and reflexive terms. That model reaches 0.95 mean Jensen–Shannon similarity with the LM's next-token distribution, against 0.44 for the one-hot positional view, which falls below the 0.5 uniform baseline. The pattern holds across nine models (Gemma-2, Qwen2.5, Llama-3.1; 2–72B) and ten tasks, and it persists when up to 10,000 filler tokens are interleaved. Binding retrieval is therefore a position-weighted ensemble. Weakening lexical signal against a noisier positional one is offered as a mechanistic account of lost-in-the-middle.
Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.
More papers — OpenAlex / S2
Affiliations (1)
- Tel Aviv University(institute)
Co-authors (12)
- Atticus Geiger15 shared
- Yoav Gur-Arieh9 shared
- Lee Sharkey6 shared
- Max Tegmark6 shared
- Neel Nanda6 shared
- Adrià Garriga-Alonso4 shared
- Arthur Conmy4 shared
- Bilal I. Chughtai4 shared
- Casper, Stephen4 shared
- Daniel Murfet4 shared
- David Anthony Bau4 shared
- Eric J. Michaud4 shared
Their work is cited by (6)
- Model Alignment Search2× refs
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?2× refs
- Steering at the Source: Style Modulation Heads for Robust Persona Control2× refs
- Do Sparse Autoencoders Capture Concept Manifolds?2× refs
- Tracing Persona Vectors Through LLM Pretraining2× refs
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds2× refs
Other inbound relations (1)
Recent mentions (1)
- papers-typedgur-arieh-2025-mixing-mechanisms.md