paper:mechanistic-faithfulness-66c9d7A Toy Model of Mechanistic (Un)Faithfulness
TL;DR
Transcoders can learn a different computation from the layer they replace, even when the layer is trivial and the transcoder is monosemantic with low on-distribution error. In an absolute-value toy model, y = ReLU(x) + ReLU(-x), a transcoder trained with tanh-L1 sparsity recovers the expected ReLU(x_i) and ReLU(-x_i) features. Once a single point p = [1,1,1,0,...] is repeated in the training data, the transcoder grows a 'datapoint feature', y += p·ReLU(λp·x − b), that memorizes p. The underlying model has no notion of p, so the feature is mechanistically unfaithful. Memorization tracks the product of repetition frequency and sparsity penalty. A 5% repetition rate is enough to produce the feature, and the three constituent features are rotated so they no longer fire on p. The paper's instrument is a compressed-sensing decomposition of each transcoder feature's output contribution over a library of ideal circuits, plus a crude memorization metric m(f_i, p). The proposed remedy is Jacobian matching, which penalizes ||J_MLP − J_Transcoder||_F^2 and is estimated cheaply with random-vector products. It removes the datapoint feature in this example. Superposition blurs the true Jacobian, so a perfect match is not the goal. The implication is that transcoders model computation, not just representation, so faithfulness is a separate failure mode from simplicity. It cuts at the foundation of interpretability's claim to trustworthy explanations, and Jacobian-style constraints are a promising route to addressing it.
What to take away
- 1. Mechanistic faithfulness asks whether a transcoder implements the same mechanism as the layer it replaces, and it becomes a live concern because transcoders model computation rather than just representation as SAEs do.
- 2. With a 5% repeat fraction of p = [1,1,1,0,...] in the absolute-value toy problem, a transcoder learns a datapoint feature y += p·ReLU(λp·x − b) that the underlying abs(x) model does not contain.
- 3. In the no-repeat baseline, a transcoder trained on abs(x) with tanh-L1 regularization recovers almost exactly the expected ReLU(x_i) and ReLU(-x_i) features, with only a small residual.
- 4. Datapoint features form only when two conditions hold, a repeated datapoint and a sparsity penalty, and a hyperparameter scan shows that their product controls memorization.
- 5. Under light repetition, the three transcoder features covering the dimensions of p are rotated so they do not activate on p, which means the unfaithfulness distorts the generalizing features too.
- 6. The memorization metric m(f_i, p) = ReLU(y|_{f_i}(p)·p̂ − max_{j<3} y|_{f_i}(p)·e_j) counts excess explanation of p beyond what any single constituent dimension accounts for, summed over features.
- 7. The Jacobians at the memorized point differ clearly: the baseline gives a diagonal matrix (independent features), while the datapoint-feature transcoder makes each feature active because of all the others.
- 8. Adding a Jacobian-matching loss, estimated as E_v||J_MLP v − J_Transcoder v||² with random vectors v via backprop, substantially reduces the memorization metric and removes the datapoint feature in the toy example.
- 9. A replicable method is to decompose each transcoder feature's output contribution over many sampled data points against a library of ideal circuits (ReLU(x_i), ReLU(-x_i), memorization) using compressed sensing, which yields a feature-by-circuit matrix plus an unexplained residual.
- 10. Open question: transcoders may cheat Jacobian matching through gradient masking, with large-weight features that have very negative biases, and the note suggests tanh-L1 and penalizing per-feature Jacobian contributions as mitigations, while superposition means a perfect Jacobian match is undesirable.
Peer brief — for seminar discussion
Chris Olah's informal note, flagged as preliminary and low-confidence, builds a toy model of mechanistic (un)faithfulness, meaning the worry that a transcoder replacing an MLP can match its outputs while implementing a different mechanism. The setup is a one-layer absolute-value problem, y = ReLU(x) + ReLU(-x), where a transcoder with tanh-L1 sparsity cleanly recovers the expected 2n ReLU features. Then a single point p = [1,1,1,0,...] is repeated in the training data. At a 5% repeat fraction the transcoder grows a datapoint feature, y += p·ReLU(λp·x − b), which memorizes something the true model never did, and it rotates the three constituent features so they no longer fire on p. Memorization depends on the product of repetition rate and sparsity pressure. The instrument is a compressed-sensing decomposition of each transcoder feature's output contributions over a library of ideal circuits, paired with a crude memorization score. The proposed fix is Jacobian matching: penalize ||J_MLP − J_Transcoder||_F^2, estimated with random-vector products. It largely eliminates the datapoint feature, though some rotation remains in the residual. The load-bearing finding is that low MSE and monosemanticity do not guarantee mechanistic faithfulness once the objective is computation. The note also separates faithfulness from simplicity and says it is more worried about the former. It hypothesizes that Jacobians are a canonicalized view of a layer's mechanism, and that superposition blurs the Jacobian tail, so exact matching should not be the target. An alternative to Jacobian matching would be directly matching downstream behavior or logits, or weight-based interpretability methods that compare parameters. The main pushback concerns scope and evidence. The demonstration is a hand-built abs(x) model with one planted repeated point, where unfaithfulness is defined by construction because the ground-truth mechanism is known. Real models have no such oracle, and genuine memorization in them would look identical to a transcoder artifact. The memorization metric is acknowledged as crude, and the Jacobian fix is shown only on a toy problem, with no test at scale and no quantitative comparison against alternatives. Gradient masking is raised as a risk but not tested. Whether this failure appears in production transcoders, and whether Jacobian matching survives superposition, remains open.
Methods (9)
- Absolute value transcoder problemToy computation y=abs(x) implemented by two ReLU neurons, used to study unfaithfulness.
- Attribution GraphsGradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.
- Compressed-sensing ideal-circuit library visualizationMaps transcoder features onto a library of ideal circuits via compressed sensing, with residual.
- Jacobian matchingPenalizing difference between transcoder and MLP Jacobians, computed cheaply via random vector products.
- Memorization metric m(f_i,p)Crude measure of excess explanation of p over constituent dimensions, summed across features.
- Repeated memorization data point setupTraining distribution where special point p=[1,1,1,0,...] appears with fraction repeat_frac.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- tanh L1 regularizationSparsity regularizer approximating L0 used to avoid duplicate features.
- TranscodersDecomposition method for activations; VPD is compared against transcoders in sparsity-reconstruction tradeoff.
Frameworks (1)
- Two-category taxonomy of SAE/transcoder concerns (faithfulness vs simplicity)Refactoring of concerns into mechanistic faithfulness and simplicity.
Findings (7)
- Baseline Jacobian on the memorized point is diagonal, while with datapoint feature each feature's activity depends on all others
Motivates Jacobian connection.
- With 5% repeated data point, transcoder learns a datapoint feature and rotated constituent features that do not activate on p
Demonstrates unfaithfulness in a simple setup.
- Jacobian matching removes the memorization feature in repeated-data example, leaving constituent features slightly rotated
Toy-model fix of unfaithfulness.
- Without repetition, transcoder on abs problem learns almost exactly ReLU(x_i) and ReLU(-x_i) features with small residual
Baseline toy experiment.
- Datapoint feature formation is controlled by the product of repetition fraction and sparsity regularization strength
Hyperparameter scan result.
- Transcoders easily discover the ReLU(x)+ReLU(-x) solution by default on the absolute value problem
Baseline result before repetition.
- Adding Jacobian matching loss dramatically reduces datapoint-feature memorization metric
Result of Jacobian matching.
Claims (9)
- Moving from SAEs modeling representation to transcoders modeling computation makes unfaithful mechanisms possible even with monosemantic, low-MSE features
Core motivation.
- A layer's Jacobian across data points is a canonicalized form of parameters that captures mechanism and neuron co-activation
Rationale for Jacobian matching.
- Mechanistic faithfulness is the primary concern, cutting at the heart of why we care about mech interp, more than simplicity
Authors' prioritization.
- Mechanistic unfaithfulness attacks one of the deepest hopes of mech interp: explanations we can deeply trust
Stakes of the problem.
- Datapoint features are not mechanistically faithful when the underlying model has no special notion of p
Memorization by transcoder not in model.
- Because of superposition, we should not expect or want to match the Jacobian perfectly
Limit on Jacobian matching.
- One-step-at-a-time replacement bounds how far transcoder mechanisms can diverge
Partial mitigation of unfaithfulness.
- Mechanistic faithfulness can be a real problem even in very simple setups
Takeaway of toy experiments.
- Jacobian matching can solve mechanistic faithfulness in this toy example
Positive result with caveats.
Hypotheses (2)
- Transcoders could cheat Jacobian matching via large-weight, very negative-bias features exploiting the L1/L0 gap
Gradient masking risk; mitigated by tanh L1 or penalizing Jacobian contribution L1.
- Datapoint features form only if there is a repeated datapoint in training and sparsity regularization is applied
Two necessary conditions.
Questions (3)
- If we don't give a model enough neurons to implement this solution, how does it do computation in superposition?
Left to future work.
- Why should we believe our transcoders learn the same mechanisms as the original underlying model?
Motivating question.
- Is the transcoder capturing the true mechanisms?
Defines the faithfulness category.
Original abstract (expand)
This paper explores mechanistic faithfulness in sparse model approximations like transcoders, which are SAEs that replace model components but may implement different computational mechanisms than the original model. Using a toy model based on the absolute value function, the authors demonstrate how transcoders can develop spurious 'datapoint features' that memorize specific repeated inputs rather than faithfully reproducing the original model's mechanisms. The work investigates when these mechanistically unfaithful features emerge and proposes approaches like Jacobian matching to address this fundamental concern for mechanistic interpretability.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 84%
- ≈ 84%
- Toy Models of Superpositionin corpus2022≈ 84%
- Insights on Crosscoder Model Diffingin corpus≈ 84%
- Mechanistic Interpretability of Code Correctness in LLMs via Sparse AutoencodersKriz Tahimic and Charibeth Cheng2025≈ 83%
- ≈ 83%
- A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious MinimaHarshvardhan Saini, Zhaoqian Yao, Zheng Lin, Yizhen Liao, Jingyi Cui, Yisen Wang, Mengnan Du, Dianbo Liu Yiming Tang2026≈ 83%
- Inference-Time Machine Unlearning via Gated Activation RedirectionOt\'avio Parraga, Jo\~ao Vitor Boer Abitante, Kristen K. Arguello, Joana Pasquali, Ramiro N. Barros, Flavio du Pin Calmon, Christian Mattjie, Rodrigo C. Barros, Lucas S. Kupssinsk\"u Vin\'icius Conte Turani2026≈ 83%
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?in corpus2025≈ 83%
- Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable EvaluationDarian Lee, Magy Seif El-Nasr Reza Habibi2026≈ 83%
- Differentiable Faithfulness Alignment for Cross-Model Circuit TransferBinxu Wang, Shay B. Cohen, Anna Korhonen, Yonatan Belinkov Shun Shao2026≈ 83%
- ≈ 83%
- Constructing Interpretable Features from Compositional Neuron GroupsAtticus Geiger, Mor Geva Or Shafran2026≈ 83%
- Measuring Faithfulness in Chain-of-Thought Reasoningin corpus2023≈ 82%
- Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation AnalysisAshish Kattamuri, Rahul Raja, Arpita Vats, Ishita Prasad, Akshata Kishore Moharir Harshwardhan Fartale2026≈ 82%
- Reasoning Resides in Layers: Restoring Temporal Reasoning in Video-Language Models with Layer-Selective MergingHaonan Wang, Jian Kang, Kenji Kawaguchi, Jiaying Wu Zihang Fu2026≈ 82%
- What do Language Models Learn and When? The Implicit Curriculum HypothesisKaiser Sun, Millicent Li, Isabelle Lee, Lindia Tjuatja, Jen-tse Huang, Graham Neubig Emmy Liu2026≈ 82%
- ≈ 82%
- ≈ 82%
- ≈ 82%
- ≈ 82%
- Reasoning Models Don't Always Say What They Thinkin corpus2025≈ 82%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 82%
- ≈ 82%
- ≈ 82%
- ≈ 82%
- Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representationsSadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji Sanjay Basu2026≈ 82%
- Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label PairsAlex McKenzie, Florin Pop, Stijn Servaes, Martin Leitgab, Mike Vaiana, Judd Rosenblatt, Michael S. A. Graziano, Diogo de Lucena Keenan Pepper2026≈ 82%
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentionin corpus2026≈ 82%