paper:replicating-circuit-2e7989Replicating Circuit Tracing for a Simple Known Mechanism - Goodfire
TL;DR
Cross-Layer Transcoders (CLTs) trained on GPT-2 Small recover interpretable greater-than machinery from a single prompt, but the recovered circuit is messier and more lookup-like than the MLP-neuron circuit mapped by Hanna et al. (2023). The CLT used 49,152 features per layer (589,824 total, expansion factor 64) with ReLU activations, trained on 100M FineWeb tokens, reaching L0 of 79 and 59% token accuracy. An attribution graph for 'The war lasted from the year 1711 to 17', built with a reimplementation of Ameisen et al.'s construction and pruning, yielded 160 pre-deduplication features of interest (top 16 features for each of the top 10 logits). Replicating Hanna et al.'s logit-attribution heatmaps over ~100 input years, with decoder vectors summed across all later layers, showed layer 8-9 features such as 425104, 407684 and 451291 promoting large years, alongside narrow features (461858 active only for inputs 6-14; 349410 only for input 11), an inverse feature (402486), and unexpected structured-number features: a parity feature (399423) and a multiples-of-5 feature (456115). Heatmaps and max-activating examples often disagree. CLTs find relevant structure without Hanna et al.'s hand-built interventions, yet sparse replacement models may decompose numerical computation differently, closer to the heuristic features in Lindsey et al.'s addition circuits. Multi-prompt feature selection and cross-position tracing remain untested.
What to take away
- 1. The CLT trained on 100M FineWeb tokens for GPT-2 Small uses an expansion factor of 64, giving 49,152 features per layer and 589,824 across 12 layers, and reaches L0 of 79 with 59% token accuracy, in line with Ameisen et al.'s reported numbers.
- 2. Graph construction used a single prompt ('The war lasted from the year 1711 to 17'), ranking edges to the top 10 logits and keeping the top 16 features per logit to give 160 features of interest before deduplication.
- 3. Layer 8 feature 425104, the top input feature to the 18 logit, activates on the larger years in dense chronological contexts (even Roman-numeral dates), but its heatmap promotes outputs above about 60 almost independently of the input year, unlike Hanna et al.'s input-dependent neurons.
- 4. Feature 461858 (layer 9) has nonzero attribution only for input years 6 to 14 and promotes outputs from about 10 to 30 before suppressing larger ones, and feature 349410 (layer 7) has nonzero attribution only for input year 11, which makes both look like special-case lookups instead of a general comparator.
- 5. Feature 402486 ranks 14th of 137 features yet implements an inverse greater-than, activating only for inputs under 27 and promoting low outputs, and its max-activating examples suggest it is a general 'small number contexts' detector.
- 6. CLTs surfaced a parity feature (399423, ranked 6th of 146 for logit 20) that promotes even outputs for odd inputs, and its max-activating examples are contrastive pairs such as 'pros and cons' and 'right or wrong', which suggests an abstract opposition representation.
- 7. A multiples-of-5 feature (456115) promotes outputs ending in 5 or 0 and suppresses those ending in 4, 6, 9 and 1, and feature 475549 maps inputs ending in 1 to outputs ending in 2, so many high-ranked features are structured-number heuristics with no analogue in Hanna et al.'s circuit.
- 8. Methodologically, features are selected on one prompt and then interpreted with a years-by-years logit-attribution matrix, built by running the local replacement model on about 100 prompts 'The war lasted from the year 17YY to 17' and summing each feature's decoder vectors across all later layers (v = Σ a·W_dec).
- 9. Heatmaps and max-activating examples disagree for many features (425104 looks input-independent in logit attribution but date-dependent in examples, and 349410 shows the reverse), and the authors speculate that isolated logit attribution ignores feature interactions and was validated only on Transformers, not on CLT features that write to multiple layers.
- 10. Open question: would aggregating feature scores over many prompts remove the distractor features and give a cleaner circuit closer to Hanna et al.'s, and how do CLT number features relate to the helical number representations of Kantamneni & Tegmark?
Peer brief — for seminar discussion
The work trains a Cross-Layer Transcoder on GPT-2 Small and tests whether it can recover the well-studied greater-than circuit of Hanna et al. (2023). The CLT has 49,152 features per layer (589,824 total, expansion factor 64, ReLU rather than JumpReLU), is trained on 100M FineWeb tokens, and reaches L0 of 79 and 59% token accuracy. The method is attribution-graph analysis on one prompt, 'The war lasted from the year 1711 to 17'. The top 16 features for each of the top 10 logits give 160 candidates. Each candidate is then examined with a logit-attribution heatmap, computed over about 100 input years by summing the feature's decoder vectors across all later layers, plus max-activating examples. The load-bearing finding is that CLTs surface relevant, interpretable features without hand-designed interventions. These include layer 8-9 greater-than features (425104, 407684, 451291), narrow-range features (461858 for inputs 6-14, 349410 for input 11 only), an inverse feature (402486), and surprising structured-number features such as a parity feature (399423) and a multiples-of-5 feature (456115). Yet the picture is less clean than Hanna et al.'s. Several top features show little input dependence, and heatmaps often contradict max-activating examples. The implication is that sparse replacement models may decompose numerical computation into many heuristic, lookup-like pieces, reminiscent of the two-digit addition features in Lindsey et al., instead of a few monotone comparators. The stated hypothesis is that multi-prompt feature aggregation would remove distractors and yield a circuit closer to the original. The main contestable point is that the evidence is qualitative and hand-curated. Features were chosen from one prompt, then hand-selected for 'clear or interesting' interpretations, with no quantitative measure of how much of the greater-than behavior the CLT circuit explains. Nothing is ablated or patched, so there is no causal test. The authors admit that logit attribution, built for Transformer components, may not transfer to CLT features that write to multiple layers. That gap could produce the very divergences from Hanna et al. that they report. A sceptic would also say the title's 'replication' is generous, since the paper compares heatmap shapes and never tests faithfulness. An alternative would be to run activation patching or ablation on the CLT features, or to use a standard per-layer sparse autoencoder or transcoder baseline, to separate CLT-specific effects from generic dictionary-learning artifacts. The reading of the parity feature as abstract opposition is also speculative, resting on a few examples.
Methods (9)
- Attribution GraphsGradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.
- Full replacement modelSubstitutes all MLP outputs with CLT reconstructions.
- Local replacement modelCLT plus error nodes and original attention patterns, local to a prompt.
- Logit attribution analysisDot product of summed decoder vector with unembedding vector, following Hanna et al.
- Max-activating examplesUsed to interpret features alongside heatmaps.
- Summed decoder vector across later layersAdaptation of logit attribution to CLT features that decode into all subsequent layers.
- SupernodesManual grouping of related features to simplify attribution graphs.
- Top-16 features per top-10 logits selectionYields 160 features of interest from a single prompt.
- Years-by-years logit attribution heatmapsInput-output mapping per feature across ~100 prompts.
Frameworks (1)
- Cross-Layer TranscodersMLP replacement with sparse features underlying the original attribution graphs.
Datasets (1)
- Greater-than task promptsTemplate prompts; 1711 for graph, ~100 for analysis.
Findings (16)
- Feature 475549 activates on inputs ending in 1 and promotes tokens ending in 2; max-activating on tokens preceding 2 in lists and timestamps
Structured number feature.
- Feature 451291 promotes years above 39 with input dependence, zero attribution for some inputs, and local pockets of promotion/suppression
More complex than simple comparison.
- Feature 399423 (rank 6 of 146 for logit 20) promotes even outputs for odd inputs; max-activating on contrastive pairs like 'pros and cons'
Abstract parity feature.
- Feature 402486 (rank 14 of 137) implements an inverse greater-than: active for inputs under 27, promotes low outputs and suppresses larger
Unexpected behavior; may be general small-number detector.
- Attribution graph for the 1711 prompt shows dense feature concentration from token 11 to final position, concentrated in later layers
Structure of the computational pathway.
- Feature 349410 has non-zero attribution only for input year 11 promoting outputs around 11, yet activates on broader number contexts
Special-case lookup-like feature.
- Feature 425104 (layer 8) is top feature for logit 18; activates on larger years in chronological contexts, including Roman numerals
Greater-than feature.
- Feature 407684 promotes predicted years above ~15 and its max-activating examples extend to prices, percentages, measurements
More general greater-than feature.
- Feature 456115 promotes multiples of 5 and suppresses numbers ending in 4, 6, 9, 1; activates on punctuation near token 5
Structured number feature.
- CLT heatmaps show broad similarity to Hanna et al. in columnar structure but less input dependence for top features
Comparison of heatmaps.
Claims (10)
- Logit attribution techniques validated in Transformers may not transfer directly to CLTs since each feature writes multiple vectors
Caveat.
- CLTs reveal novel features like abstract parity detectors pointing to richer representational structures than previously recognized
Closing claim.
- Divergence between heatmaps and max-activating examples may arise because logit attribution measures a feature in isolation
Speculative explanation.
- CLTs discover interpretable and relevant features for greater-than, many recognizably similar to those of Hanna et al.
Main conclusion.
- CLT feature space is somewhat less clean than Hanna et al.'s, with some narrow special-case-lookup-like features
Difference between sparse and neuron analyses.
- Roman numeral activation indicates the model learned semantic date representations beyond character matching
Interpretation of Feature 425104.
- Feature 407684 represents a broader greater-than pattern across numerical contexts, not just chronological
Interpretation.
- CLTs uncover relevant structure out of the box without labor-intensive experimental interventions
Advantage over original analysis.
- Sparse interpreter models may represent and decompose computational mechanisms differently
Hinted by differences.
- Feature 402486 may serve as a general detector for small-number contexts
Interpretation of inverse feature.
Hypotheses (3)
- Aggregating feature scores over multiple prompts would weed out distractor features and isolate a cleaner circuit closer to Hanna et al.
Limitation of single-prompt selection.
- Many similar narrow greater-than features exist that activate for different prompts
Inferred from Feature 461858.
- Nouns with longer durations skew output to larger numbers
Proposed future study.
Questions (3)
- How do CLT number representations relate to those in non-sparse models, such as helical representations?
Future work.
- By what mechanisms does the noun in '[noun] lasted from 17YY to 17' influence the final logits?
Future work using cross-position tracing.
- Why do logit attribution heatmaps diverge from max-activating examples?
Unclear reason stated by authors.
Original abstract (expand)
We train Cross-Layer Transcoders (CLTs) on GPT-2 Small and evaluate their ability to recover known computational mechanisms using the greater-than task as a case study. CLTs successfully identify relevant and interpretable features including abstract parity features, validating their ability to semi-autonomously discover interesting structure in language models. We observe some differences from prior circuit analysis work, suggesting possible differences in how sparse interpreter models represent computational mechanisms.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution GraphsAbir Harrasse, Vedant Palit, Tung-Yu Wu, Jiarui Liu, Punya Syon Pandey, Roderick Wu, Terry Jingchen Zhang, Zhijing Jin, Bernhard Sch\"olkopf Florent Draye2026≈ 85%
- ≈ 83%
- Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination MitigationZekai Ye, Xiaocheng Feng, Weihong Zhong, Weitao Ma, Xiachong Feng Qiming Li2025≈ 83%
- Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMsin corpus2026≈ 83%
- ≈ 83%
- Protein Circuit Tracing via Cross-layer TranscodersKunal Talreja, Daniel Saeedi, Amirali Aghazadeh Darin Tsui2026≈ 83%
- ≈ 82%
- ≈ 82%
- ≈ 82%
- Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph DecompositionShahnawaz Alam, Mohammed Kaif Pasha Mohammed Mudassir Uddin2026≈ 82%
- Quantifying LLM Attention-Head Stability: Implications for Circuit UniversalityJack Stanley, Praneet Suresh, Danilo Bzdok Karan Bali2026≈ 82%
- Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language ModelsPhilip Torr, Fazl Barez Michael Lan2024≈ 81%
- Exhaustive Circuit Mapping of a Single-Cell Foundation Model Reveals Massive Redundancy, Heavy-Tailed Hub Architecture, and Layer-Dependent Differentiation ControlIhor Kendiukhov2026≈ 81%
- Circuit Complexity of Hierarchical Knowledge Tracing and Implications for Log-Precision TransformersRichard Baraniuk and Shashank Sonkar Naiming Liu2026≈ 81%
- Zoom In: An Introduction to Circuitsin corpus2020≈ 81%
- Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit AnalysisYan Hu, Wenyu Du, Reynold Cheng, Benyou Wang, Difan Zou Xu Wang2025≈ 81%
- Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language ExplanationsAjay Pravin Mahale2026≈ 81%
- Evaluating Brain-Inspired Modular Training in Automated Circuit Discovery for Mechanistic InterpretabilityJatin Nainani2024≈ 80%
- A Mathematical Framework for Transformer Circuitsin corpus2021≈ 80%
- Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language ModelsZhiyu Xue, Yihao Quan, Benlin Liu, Zeru Shi, Michelle Hurst, Jacob Feldman, Ruixiang Tang, Ranjay Krishna, Vladimir Pavlovic Liwei Che2026≈ 80%
- Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation AnalysisAshish Kattamuri, Rahul Raja, Arpita Vats, Ishita Prasad, Akshata Kishore Moharir Harshwardhan Fartale2026≈ 80%
- Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without TrainingVincent Gripon, Bastien Pasdeloup, Axel Marmoret, Lukas Mauch, Fabien Cardinaux, Ghouthi Boukli Hacene Jonathan Lys2026≈ 80%
- Insights on Crosscoder Model Diffingin corpus≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 80%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 80%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 80%
- ≈ 80%