paper
active
paper:replicating-circuit-2e7989

Replicating Circuit Tracing for a Simple Known Mechanism - Goodfire

TL;DR

Cross-Layer Transcoders (CLTs) trained on GPT-2 Small recover interpretable greater-than machinery from a single prompt, but the recovered circuit is messier and more lookup-like than the MLP-neuron circuit mapped by Hanna et al. (2023). The CLT used 49,152 features per layer (589,824 total, expansion factor 64) with ReLU activations, trained on 100M FineWeb tokens, reaching L0 of 79 and 59% token accuracy. An attribution graph for 'The war lasted from the year 1711 to 17', built with a reimplementation of Ameisen et al.'s construction and pruning, yielded 160 pre-deduplication features of interest (top 16 features for each of the top 10 logits). Replicating Hanna et al.'s logit-attribution heatmaps over ~100 input years, with decoder vectors summed across all later layers, showed layer 8-9 features such as 425104, 407684 and 451291 promoting large years, alongside narrow features (461858 active only for inputs 6-14; 349410 only for input 11), an inverse feature (402486), and unexpected structured-number features: a parity feature (399423) and a multiples-of-5 feature (456115). Heatmaps and max-activating examples often disagree. CLTs find relevant structure without Hanna et al.'s hand-built interventions, yet sparse replacement models may decompose numerical computation differently, closer to the heuristic features in Lindsey et al.'s addition circuits. Multi-prompt feature selection and cross-position tracing remain untested.

What to take away

  1. 1. The CLT trained on 100M FineWeb tokens for GPT-2 Small uses an expansion factor of 64, giving 49,152 features per layer and 589,824 across 12 layers, and reaches L0 of 79 with 59% token accuracy, in line with Ameisen et al.'s reported numbers.
  2. 2. Graph construction used a single prompt ('The war lasted from the year 1711 to 17'), ranking edges to the top 10 logits and keeping the top 16 features per logit to give 160 features of interest before deduplication.
  3. 3. Layer 8 feature 425104, the top input feature to the 18 logit, activates on the larger years in dense chronological contexts (even Roman-numeral dates), but its heatmap promotes outputs above about 60 almost independently of the input year, unlike Hanna et al.'s input-dependent neurons.
  4. 4. Feature 461858 (layer 9) has nonzero attribution only for input years 6 to 14 and promotes outputs from about 10 to 30 before suppressing larger ones, and feature 349410 (layer 7) has nonzero attribution only for input year 11, which makes both look like special-case lookups instead of a general comparator.
  5. 5. Feature 402486 ranks 14th of 137 features yet implements an inverse greater-than, activating only for inputs under 27 and promoting low outputs, and its max-activating examples suggest it is a general 'small number contexts' detector.
  6. 6. CLTs surfaced a parity feature (399423, ranked 6th of 146 for logit 20) that promotes even outputs for odd inputs, and its max-activating examples are contrastive pairs such as 'pros and cons' and 'right or wrong', which suggests an abstract opposition representation.
  7. 7. A multiples-of-5 feature (456115) promotes outputs ending in 5 or 0 and suppresses those ending in 4, 6, 9 and 1, and feature 475549 maps inputs ending in 1 to outputs ending in 2, so many high-ranked features are structured-number heuristics with no analogue in Hanna et al.'s circuit.
  8. 8. Methodologically, features are selected on one prompt and then interpreted with a years-by-years logit-attribution matrix, built by running the local replacement model on about 100 prompts 'The war lasted from the year 17YY to 17' and summing each feature's decoder vectors across all later layers (v = Σ a·W_dec).
  9. 9. Heatmaps and max-activating examples disagree for many features (425104 looks input-independent in logit attribution but date-dependent in examples, and 349410 shows the reverse), and the authors speculate that isolated logit attribution ignores feature interactions and was validated only on Transformers, not on CLT features that write to multiple layers.
  10. 10. Open question: would aggregating feature scores over many prompts remove the distractor features and give a cleaner circuit closer to Hanna et al.'s, and how do CLT number features relate to the helical number representations of Kantamneni & Tegmark?

Peer brief — for seminar discussion

The work trains a Cross-Layer Transcoder on GPT-2 Small and tests whether it can recover the well-studied greater-than circuit of Hanna et al. (2023). The CLT has 49,152 features per layer (589,824 total, expansion factor 64, ReLU rather than JumpReLU), is trained on 100M FineWeb tokens, and reaches L0 of 79 and 59% token accuracy. The method is attribution-graph analysis on one prompt, 'The war lasted from the year 1711 to 17'. The top 16 features for each of the top 10 logits give 160 candidates. Each candidate is then examined with a logit-attribution heatmap, computed over about 100 input years by summing the feature's decoder vectors across all later layers, plus max-activating examples. The load-bearing finding is that CLTs surface relevant, interpretable features without hand-designed interventions. These include layer 8-9 greater-than features (425104, 407684, 451291), narrow-range features (461858 for inputs 6-14, 349410 for input 11 only), an inverse feature (402486), and surprising structured-number features such as a parity feature (399423) and a multiples-of-5 feature (456115). Yet the picture is less clean than Hanna et al.'s. Several top features show little input dependence, and heatmaps often contradict max-activating examples. The implication is that sparse replacement models may decompose numerical computation into many heuristic, lookup-like pieces, reminiscent of the two-digit addition features in Lindsey et al., instead of a few monotone comparators. The stated hypothesis is that multi-prompt feature aggregation would remove distractors and yield a circuit closer to the original. The main contestable point is that the evidence is qualitative and hand-curated. Features were chosen from one prompt, then hand-selected for 'clear or interesting' interpretations, with no quantitative measure of how much of the greater-than behavior the CLT circuit explains. Nothing is ablated or patched, so there is no causal test. The authors admit that logit attribution, built for Transformer components, may not transfer to CLT features that write to multiple layers. That gap could produce the very divergences from Hanna et al. that they report. A sceptic would also say the title's 'replication' is generous, since the paper compares heatmap shapes and never tests faithfulness. An alternative would be to run activation patching or ablation on the CLT features, or to use a standard per-layer sparse autoencoder or transcoder baseline, to separate CLT-specific effects from generic dictionary-learning artifacts. The reading of the parity feature as abstract opposition is also speculative, resting on a few examples.

Methods (9)

Frameworks (1)

Datasets (1)

Findings (16)

Original abstract (expand)

We train Cross-Layer Transcoders (CLTs) on GPT-2 Small and evaluate their ability to recover known computational mechanisms using the greater-than task as a case study. CLTs successfully identify relevant and interpretable features including abstract parity features, validating their ability to semi-autonomously discover interesting structure in language models. We observe some differences from prior circuit analysis work, suggesting possible differences in how sparse interpreter models represent computational mechanisms.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar