paper:doi-10-48550-arxiv-2501-16496Open Problems in Mechanistic Interpretability
TL;DR
Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.
What to take away
- 1. A sparse autoencoder with 16 million latents inserted into GPT-4 yields language-modeling loss equal to a model trained with only about 10% of GPT-4's pretraining compute, which shows SDL reconstruction error is far from negligible.
- 2. Makelov et al. found that substituting SAE reconstructions into GPT-2 small cut performance by 10% when the SAE was trained on task-specific data and by 40% when trained on the full distribution.
- 3. Engels et al. showed that much of the direction and norm of SAE reconstruction error is linearly predictable from the input activation, so the residual is structured and not noise, which suggests systematic blind spots but also fixes.
- 4. Kissane et al. found that SAEs trained on pretraining data lack good latents for the concept of refusing harmful requests, while SAEs trained on chat-formatted data have them, so the latents SDL finds depend on its training distribution.
- 5. Miller et al. showed that circuit faithfulness metrics depend on the causal-intervention implementation and that within-task variance is large, so circuits capture average-case but poorly individual-datapoint behavior.
- 6. The review proposes a three-step reverse-engineering framework of decomposition, description and validation, contrasted with concept-based interpretability, and runs circuit discovery (task definition, DAG decomposition, patching, description-validation loop, final validation by faithfulness, minimality and completeness) as a worked pipeline.
- 7. A replicable methodology choice is validating interpretability tools on models with known ground truth, such as programs compiled into weights (Tracr) or models trained with Interchange Intervention Training, as in Conmy et al.'s ACDC test on a model implementing a known algorithm.
- 8. Mechanistic interpretability research is concentrated on three model families (CNN image models, BERT and GPT), and the transfer of methods to state space models like Mamba is only partly shown (Paulo et al. for CNN to SSM; Ensign and Garriga-Alonso for IOI circuits in Mamba).
- 9. Open question: whether the superposition hypothesis is fundamentally valid or only pragmatically useful, and whether only local feature geometry or the full global geometry must be understood, which would determine whether a bag-of-features SDL approach can suffice.
- 10. Finetuning appears to make shallow edits to existing circuits (Prakash et al., Jain et al., Lee et al.), so harmlessness training can be cheaply undone, and Arditi et al. found a single refusal direction that Lin et al. used to red-team models.
Peer brief — for seminar discussion
This is a forward-looking synthesis by 27 authors, coordinated by Apollo Research for Schmidt Sciences, that maps what mechanistic interpretability still cannot do. It does not report new experiments. It organizes the field into methods and foundations, applications, and socio-technical problems, and it ends with an appendix of several hundred numbered open questions. The central scaffold is a three-step reverse-engineering framework of decomposition, description and validation, set against concept-based interpretability such as probes. Circuit discovery serves as the worked case study of a full pipeline. The load-bearing finding is that the leading decomposition method, sparse dictionary learning, is not yet faithful or conceptually settled. A 16M-latent SAE spliced into GPT-4 degrades loss to that of a model with about 10% of the compute. Makelov et al. report GPT-2 small performance drops of 10% and 40%, depending on the SAE's training distribution. Error nodes do not fix this, since they absorb everything else. Further problems follow: dependence on the linear representation hypothesis, sparsity as a doubtful proxy, feature splitting and absorption, unexplained geometry, and dataset-dependent latents. Circuit faithfulness is likewise fragile, since Miller et al. show it varies with the intervention implementation. The implication is that near-term value lies in applications that tolerate shallow descriptions: white-box evaluations, mechanistic anomaly detection, red-teaming through the refusal direction, and steering. Deeper goals such as unlearning, editing and formal verification need better decomposition, validation on model organisms, and non-cherry-picked benchmarks. A recurring prescription is to compare against black-box baselines, because interpretability tools have rarely beaten them. The paper also names hypotheses it leaves open, including whether the universality hypothesis holds across architectures and whether superposition is a true description or a useful fiction. A critical reader would push on scope and evidentiary weight. As a consensus synthesis, it blends views that the authors say do not reflect any individual's position, so its claims about priorities are not falsifiable. Much of the critique of SDL rests on 2024 results from GPT-2 small and GPT-4-scale SAEs that newer methods such as transcoders and crosscoders may already be changing. The review also has no systematic literature-selection method, and its sources include many blog posts that are not peer reviewed. It leans on reconstruction loss as the measure of faithfulness without showing that this tracks downstream usefulness. An alternative to its survey approach would have been a quantitative meta-analysis or a pre-registered benchmark suite comparing SDL, probes and attribution methods against baselines.
Methods (22)
- Activation patching / causal interventionsSubstitute activations to observe effects.
- Activation SteeringCausal intervention technique: edit NLA explanation, reconstruct via AR, use difference as steering vector to manipulate model behavior.
- Attribution methodsMeasure causal importance of upstream variables.
- Attribution patching / AtP*Gradient-based approximation to activation patching.
- Automated Circuit DisCovery (ACDC)Automates identifying task-relevant subgraphs.
- Causal ScrubbingMethod by Chan et al. 2022 for rigorously testing interpretability hypotheses via interventions
- Concept-based probesClassifiers trained to predict concepts from activations.
- Concept bottleneck modelsIntrinsically interpretable models using concepts in training.
- Faithfulness, minimality, completenessThree attributes for evaluating circuits.
- Feature synthesisSynthesizing inputs maximizing component activation.
- Ground-truth interpretability benchmarksBenchmarks using models with known explanations.
- Highly activating dataset examplesSimplest method to describe causes of activation.
- Logit LensUnsupervised interpretability technique that projects activations through unembedding matrix; provides comparison point for NLA approach.
- Mechanistic anomaly detectionFlag actions taken for abnormal reasons.
- Model (knowledge) editing / ROMEPrecise knowledge modification; ROME has flaws.
- Path PatchingMethod by Goldowsky-Dill et al. 2023 for localizing model behavior via targeted activation interventions
- PCA / SVD dimensionality reductionEarlier decomposition methods no longer predominant.
- Reading chain-of-thoughtBehavior-based explanation that is not fully faithful.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- Sparse Dictionary LearningGeneral method for finding overcomplete sparse decompositions; the paper uses sparse autoencoders as an approximation
- TranscodersDecomposition method for activations; VPD is compared against transcoders in sparsity-reconstruction tradeoff.
- Tuned lensBaseline lens that skips ahead to output.
Frameworks (10)
- Automated InterpretabilityMethod using large language models (Claude) to generate and test explanations of features at scale
- Circuit discovery pipelineFive-step pipeline: task definition, decomposition, description, description-validation loop, final validation.
- Concept-based interpretabilitySecond approach: propose concepts and find components corresponding to them.
- CrosscodersLearn a shared feature space across two models; basis of the work.
- Distributed Alignment Search (DAS)Practical method by Geiger et al. for finding distributed causal abstractions using gradient descent
- Linear Representation HypothesisThe hypothesis that models internalize concepts as approximately linear directions in representation space; used to interpret MDS injection behavior
- PatchscopesUnifying framework for inspecting hidden representations of language models via representation interventions
- Reverse engineering (decomposition, description, validation)Three-step approach to identify roles of network components.
- Singular learning theoryRelates parameters to loss via landscape geometry.
- Superposition HypothesisCore theoretical framework: neural networks represent more features than neurons by encoding features as directions in superposition
Findings (11)
- Inserting a 16M-latent sparse dictionary into GPT-4 yields language modeling loss equivalent to a model with 10% of GPT-4's pretraining compute
Shows SDL reconstruction error is too high.
- Finetuning primarily makes shallow edits to existing representations and circuits, so harmlessness training is cheaply undone
Supported by further finetuning and causal interventions.
- Within-task variance of model performance is large: circuits approximate average case but poorly individual data points
Undermines concept-based task definition in circuit discovery.
- SAEs trained on pretraining data lack good latents for refusal, while SAEs trained on chat-formatted data have them
SAE latents depend on training data.
- SAE reconstructions decreased GPT-2 small performance by 10% (task-specific data) and 40% (full distribution)
Quantifies SAE reconstruction damage.
- A small transformer solved modular addition via a Fourier transform strategy understood only in retrospect
Alien algorithms example.
- Some gradient-based attribution methods give attributions independent of model and data-generating process
Adebayo et al. sanity checks.
- SAE reconstruction error direction and norm are largely linearly predictable from the activation vector
Errors are structured not random.
- Minimizing projection onto the refusal direction yields efficient red-teaming inputs
Interpretability insight accelerates red-teaming.
- A single SAE latent was more accurate than official dataset labels on GLUE CoLA
Example where SAE yields useful latent.
Claims (20)
- The weak linear representation hypothesis is supported by probes, steering and SAEs, while the strong version is false for some models
Position on LRH.
- Interpretability's comparative advantage lies in auditing unsafe cognition, debugging and monitoring rather than steering behavior
Prioritization argument.
- Humans and AI models often solve problems with different representations and heuristics, so reverse engineering is necessary
Motivation for reverse engineering.
- Absence of paradigmatic clarity is a major socio-technical reason interpretability has produced few real-world safety tools
Social/philosophical problem.
- Model organisms would facilitate hypothesis validation but the community lacks consensus on which networks
Proposal.
- The success of activation steering can be thought of as evidence for the linear representation hypothesis
Steering derived from LRH.
- SDL has substantial practical and conceptual limitations despite being the leading decomposition method
Core assessment of SDL.
- Probes detect correlations rather than causal variables and should be used only to generate hypotheses
Probe limitation.
- Current decomposition methods lack solid theoretical foundations and the nature of features is unclear
Calls for conceptual clarity.
- Networks do not naturally decompose into architectural components such as neurons, heads or layers
Motivates new decomposition.
Hypotheses (3)
- Finetuning improves capabilities primarily by enhancing existing circuits rather than creating new mechanisms
Prakash et al. evidence.
- SDL reconstruction error might be reduced by improved SDL training or new decomposition methods
Structured errors imply potential solutions.
- Different neural networks learn similar features and circuits to one another
Universality hypothesis to be tested across families.
Questions (10)
- Will circuit discovery provide insights into arbitrary tasks, or only where tasks can be crisply defined?
Circuit discovery scope.
- How can we reliably distinguish causal from merely correlated features when probing neural networks?
Probing open question.
- Is it possible to prove the absence of specific dangerous capabilities through mechanistic analysis?
Safety claims.
- How important is the geometry of activation space for explaining neural network behavior?
Open question A.1.1a.5.
- What should the Drosophila melanogaster of mechanistic interpretability be?
Model organism choice.
- How should we decompose networks into more interpretable constituent parts?
Central open question A.1.1a.1.
- Can current toy model verification approaches scale to frontier systems?
Formal verification open question.
- How true is the linear representation hypothesis?
Open question A.1.1a.2.
- Can the problems with SDL be overcome?
Open question A.1.1a.4.
- What is interpretability?
Socio-philosophical open question.
Original abstract (expand)
Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious MinimaHarshvardhan Saini, Zhaoqian Yao, Zheng Lin, Yizhen Liao, Jingyi Cui, Yisen Wang, Mengnan Du, Dianbo Liu Yiming Tang2026≈ 89%
- Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEsAashiq Muhamed, Yujia Zheng, Lingjing Kong, Zeyu Tang, Mona T. Diab, Virginia Smith, Kun Zhang Xiangchen Song2025≈ 87%
- Constructing Interpretable Features from Compositional Neuron GroupsAtticus Geiger, Mor Geva Or Shafran2026≈ 87%
- Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph DecompositionShahnawaz Alam, Mohammed Kaif Pasha Mohammed Mudassir Uddin2026≈ 87%
- Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representationsSadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji Sanjay Basu2026≈ 86%
- Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPTXuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, Xipeng Qiu Zhengfu He2024≈ 86%
- Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language ExplanationsAjay Pravin Mahale2026≈ 86%
- Mechanistic interpretability of large language models with applications to the financial services industryKhashayar Filom, and Arjun Ravi Kannan Ashkan Golgoon2024≈ 86%
- ≈ 86%
- Mechanistic Interpretability as Statistical Estimation: A Variance AnalysisFran\c{c}ois Portet, Maxime Peyrard Maxime M\'eloux2026≈ 86%
- Adaptive Circuit Behavior and Generalization in Mechanistic InterpretabilitySankaran Vaidyanathan, AJ Yeung, Kartik Gupta, David Jensen Jatin Nainani2024≈ 86%
- Mechanistic Interpretability of Code Correctness in LLMs via Sparse AutoencodersKriz Tahimic and Charibeth Cheng2025≈ 86%
- Mechanistic Interpretability of Diffusion Models: Circuit-Level Analysis and Causal ValidationDip Roy2026≈ 85%
- ≈ 85%
- Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable EvaluationDarian Lee, Magy Seif El-Nasr Reza Habibi2026≈ 85%
- Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future DirectionsUsman Naseem2026≈ 85%
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?in corpus2025≈ 85%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 84%
- ≈ 84%
- ≈ 84%
- ≈ 83%
- ≈ 83%
- Interpreting Language Model Parametersin corpus2026≈ 82%
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalin corpus2026≈ 82%
- ≈ 82%
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentionin corpus2026≈ 82%
- ≈ 81%
Similar preprints — Semantic Scholar
Cited by (6)
- Do Sparse Autoencoders Capture Concept Manifolds?
Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse
- Steering at the Source: Style Modulation Heads for Robust Persona Control
Residual-stream activation steering reliably degrades text coherency when steering vectors push models toward out-of-distribution behavior, and this collapse goes undetected by standard benchmarks: MM
- Tracing Persona Vectors Through LLM Pretraining
- Model Alignment Search
Model Alignment Search (MAS) establishes bidirectional causal similarity between neural networks by learning a per-model orthogonal rotation matrix that isolates behaviorally relevant subspaces and us
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
Under arbitrarily powerful alignment maps, causal abstraction becomes vacuous: any neural network can be perfectly mapped to any algorithm, a result proven formally in Theorem 1 under five mild assump