paper
active
paper:mechanistic-faithfulness-66c9d7

A Toy Model of Mechanistic (Un)Faithfulness

TL;DR

Transcoders can learn a different computation from the layer they replace, even when the layer is trivial and the transcoder is monosemantic with low on-distribution error. In an absolute-value toy model, y = ReLU(x) + ReLU(-x), a transcoder trained with tanh-L1 sparsity recovers the expected ReLU(x_i) and ReLU(-x_i) features. Once a single point p = [1,1,1,0,...] is repeated in the training data, the transcoder grows a 'datapoint feature', y += p·ReLU(λp·x − b), that memorizes p. The underlying model has no notion of p, so the feature is mechanistically unfaithful. Memorization tracks the product of repetition frequency and sparsity penalty. A 5% repetition rate is enough to produce the feature, and the three constituent features are rotated so they no longer fire on p. The paper's instrument is a compressed-sensing decomposition of each transcoder feature's output contribution over a library of ideal circuits, plus a crude memorization metric m(f_i, p). The proposed remedy is Jacobian matching, which penalizes ||J_MLP − J_Transcoder||_F^2 and is estimated cheaply with random-vector products. It removes the datapoint feature in this example. Superposition blurs the true Jacobian, so a perfect match is not the goal. The implication is that transcoders model computation, not just representation, so faithfulness is a separate failure mode from simplicity. It cuts at the foundation of interpretability's claim to trustworthy explanations, and Jacobian-style constraints are a promising route to addressing it.

What to take away

  1. 1. Mechanistic faithfulness asks whether a transcoder implements the same mechanism as the layer it replaces, and it becomes a live concern because transcoders model computation rather than just representation as SAEs do.
  2. 2. With a 5% repeat fraction of p = [1,1,1,0,...] in the absolute-value toy problem, a transcoder learns a datapoint feature y += p·ReLU(λp·x − b) that the underlying abs(x) model does not contain.
  3. 3. In the no-repeat baseline, a transcoder trained on abs(x) with tanh-L1 regularization recovers almost exactly the expected ReLU(x_i) and ReLU(-x_i) features, with only a small residual.
  4. 4. Datapoint features form only when two conditions hold, a repeated datapoint and a sparsity penalty, and a hyperparameter scan shows that their product controls memorization.
  5. 5. Under light repetition, the three transcoder features covering the dimensions of p are rotated so they do not activate on p, which means the unfaithfulness distorts the generalizing features too.
  6. 6. The memorization metric m(f_i, p) = ReLU(y|_{f_i}(p)·p̂ − max_{j<3} y|_{f_i}(p)·e_j) counts excess explanation of p beyond what any single constituent dimension accounts for, summed over features.
  7. 7. The Jacobians at the memorized point differ clearly: the baseline gives a diagonal matrix (independent features), while the datapoint-feature transcoder makes each feature active because of all the others.
  8. 8. Adding a Jacobian-matching loss, estimated as E_v||J_MLP v − J_Transcoder v||² with random vectors v via backprop, substantially reduces the memorization metric and removes the datapoint feature in the toy example.
  9. 9. A replicable method is to decompose each transcoder feature's output contribution over many sampled data points against a library of ideal circuits (ReLU(x_i), ReLU(-x_i), memorization) using compressed sensing, which yields a feature-by-circuit matrix plus an unexplained residual.
  10. 10. Open question: transcoders may cheat Jacobian matching through gradient masking, with large-weight features that have very negative biases, and the note suggests tanh-L1 and penalizing per-feature Jacobian contributions as mitigations, while superposition means a perfect Jacobian match is undesirable.

Peer brief — for seminar discussion

Chris Olah's informal note, flagged as preliminary and low-confidence, builds a toy model of mechanistic (un)faithfulness, meaning the worry that a transcoder replacing an MLP can match its outputs while implementing a different mechanism. The setup is a one-layer absolute-value problem, y = ReLU(x) + ReLU(-x), where a transcoder with tanh-L1 sparsity cleanly recovers the expected 2n ReLU features. Then a single point p = [1,1,1,0,...] is repeated in the training data. At a 5% repeat fraction the transcoder grows a datapoint feature, y += p·ReLU(λp·x − b), which memorizes something the true model never did, and it rotates the three constituent features so they no longer fire on p. Memorization depends on the product of repetition rate and sparsity pressure. The instrument is a compressed-sensing decomposition of each transcoder feature's output contributions over a library of ideal circuits, paired with a crude memorization score. The proposed fix is Jacobian matching: penalize ||J_MLP − J_Transcoder||_F^2, estimated with random-vector products. It largely eliminates the datapoint feature, though some rotation remains in the residual. The load-bearing finding is that low MSE and monosemanticity do not guarantee mechanistic faithfulness once the objective is computation. The note also separates faithfulness from simplicity and says it is more worried about the former. It hypothesizes that Jacobians are a canonicalized view of a layer's mechanism, and that superposition blurs the Jacobian tail, so exact matching should not be the target. An alternative to Jacobian matching would be directly matching downstream behavior or logits, or weight-based interpretability methods that compare parameters. The main pushback concerns scope and evidence. The demonstration is a hand-built abs(x) model with one planted repeated point, where unfaithfulness is defined by construction because the ground-truth mechanism is known. Real models have no such oracle, and genuine memorization in them would look identical to a transcoder artifact. The memorization metric is acknowledged as crude, and the Jacobian fix is shown only on a toy problem, with no test at scale and no quantitative comparison against alternatives. Gradient masking is raised as a risk but not tested. Whether this failure appears in production transcoders, and whether Jacobian matching survives superposition, remains open.

Methods (9)

Frameworks (1)

Original abstract (expand)

This paper explores mechanistic faithfulness in sparse model approximations like transcoders, which are SAEs that replace model components but may implement different computational mechanisms than the original model. Using a toy model based on the absolute value function, the authors demonstrate how transcoders can develop spurious 'datapoint features' that memorize specific repeated inputs rather than faithfully reproducing the original model's mechanisms. The work investigates when these mechanistically unfaithful features emerge and proposes approaches like Jacobian matching to address this fundamental concern for mechanistic interpretability.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar