paper
active
2025
ⓒ 11
paper:doi-10-48550-arxiv-2501-16496

Open Problems in Mechanistic Interpretability

TL;DR

Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.

What to take away

  1. 1. A sparse autoencoder with 16 million latents inserted into GPT-4 yields language-modeling loss equal to a model trained with only about 10% of GPT-4's pretraining compute, which shows SDL reconstruction error is far from negligible.
  2. 2. Makelov et al. found that substituting SAE reconstructions into GPT-2 small cut performance by 10% when the SAE was trained on task-specific data and by 40% when trained on the full distribution.
  3. 3. Engels et al. showed that much of the direction and norm of SAE reconstruction error is linearly predictable from the input activation, so the residual is structured and not noise, which suggests systematic blind spots but also fixes.
  4. 4. Kissane et al. found that SAEs trained on pretraining data lack good latents for the concept of refusing harmful requests, while SAEs trained on chat-formatted data have them, so the latents SDL finds depend on its training distribution.
  5. 5. Miller et al. showed that circuit faithfulness metrics depend on the causal-intervention implementation and that within-task variance is large, so circuits capture average-case but poorly individual-datapoint behavior.
  6. 6. The review proposes a three-step reverse-engineering framework of decomposition, description and validation, contrasted with concept-based interpretability, and runs circuit discovery (task definition, DAG decomposition, patching, description-validation loop, final validation by faithfulness, minimality and completeness) as a worked pipeline.
  7. 7. A replicable methodology choice is validating interpretability tools on models with known ground truth, such as programs compiled into weights (Tracr) or models trained with Interchange Intervention Training, as in Conmy et al.'s ACDC test on a model implementing a known algorithm.
  8. 8. Mechanistic interpretability research is concentrated on three model families (CNN image models, BERT and GPT), and the transfer of methods to state space models like Mamba is only partly shown (Paulo et al. for CNN to SSM; Ensign and Garriga-Alonso for IOI circuits in Mamba).
  9. 9. Open question: whether the superposition hypothesis is fundamentally valid or only pragmatically useful, and whether only local feature geometry or the full global geometry must be understood, which would determine whether a bag-of-features SDL approach can suffice.
  10. 10. Finetuning appears to make shallow edits to existing circuits (Prakash et al., Jain et al., Lee et al.), so harmlessness training can be cheaply undone, and Arditi et al. found a single refusal direction that Lin et al. used to red-team models.

Peer brief — for seminar discussion

This is a forward-looking synthesis by 27 authors, coordinated by Apollo Research for Schmidt Sciences, that maps what mechanistic interpretability still cannot do. It does not report new experiments. It organizes the field into methods and foundations, applications, and socio-technical problems, and it ends with an appendix of several hundred numbered open questions. The central scaffold is a three-step reverse-engineering framework of decomposition, description and validation, set against concept-based interpretability such as probes. Circuit discovery serves as the worked case study of a full pipeline. The load-bearing finding is that the leading decomposition method, sparse dictionary learning, is not yet faithful or conceptually settled. A 16M-latent SAE spliced into GPT-4 degrades loss to that of a model with about 10% of the compute. Makelov et al. report GPT-2 small performance drops of 10% and 40%, depending on the SAE's training distribution. Error nodes do not fix this, since they absorb everything else. Further problems follow: dependence on the linear representation hypothesis, sparsity as a doubtful proxy, feature splitting and absorption, unexplained geometry, and dataset-dependent latents. Circuit faithfulness is likewise fragile, since Miller et al. show it varies with the intervention implementation. The implication is that near-term value lies in applications that tolerate shallow descriptions: white-box evaluations, mechanistic anomaly detection, red-teaming through the refusal direction, and steering. Deeper goals such as unlearning, editing and formal verification need better decomposition, validation on model organisms, and non-cherry-picked benchmarks. A recurring prescription is to compare against black-box baselines, because interpretability tools have rarely beaten them. The paper also names hypotheses it leaves open, including whether the universality hypothesis holds across architectures and whether superposition is a true description or a useful fiction. A critical reader would push on scope and evidentiary weight. As a consensus synthesis, it blends views that the authors say do not reflect any individual's position, so its claims about priorities are not falsifiable. Much of the critique of SDL rests on 2024 results from GPT-2 small and GPT-4-scale SAEs that newer methods such as transcoders and crosscoders may already be changing. The review also has no systematic literature-selection method, and its sources include many blog posts that are not peer reviewed. It leans on reconstruction loss as the measure of faithfulness without showing that this tracks downstream usefulness. An alternative to its survey approach would have been a quantitative meta-analysis or a pre-registered benchmark suite comparing SDL, probes and attribution methods against baselines.

Methods (22)

Frameworks (10)

Findings (11)

Original abstract (expand)

Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar

Cited by (6)