paper:biology-language-3839a7On the Biology of a Large Language Model
TL;DR
Claude 3.5 Haiku carries out multi-step computation inside a single forward pass, and the mechanisms can be traced: given 'A rhyming couplet: He saw a carrot and had to grab it,' the model activates 'rabbit' and 'habit' planning features on the newline token before the second line begins, and clamping them rewrites the whole line toward a different rhyme. The instrument is the attribution graph, built on a local replacement model with a 30-million-feature cross-layer transcoder (CLT). Nodes are features, edges are causal interactions, and manually grouped 'supernodes' are validated by inhibition and swap interventions. Injecting 'rabbit' or 'green' planning features into 25 poems made the line end on the injected word in 70% of cases. In the Dallas prompt, swapping the Texas cluster for California makes the model output Sacramento, which shows an intermediate two-hop step. Multilingual features account for a larger share of representations in Haiku than in a smaller model, and 20 of 27 features in the multilingual nodes fire across English, French and Chinese. Hallucination follows from a default 'can't answer' circuit that 'known entity' features inhibit and that can misfire. A reward-model-bias feature fires in 100 of 100 Human/Assistant contexts versus 1 of 100 without the formatting. Because graphs gave satisfying insight on only about a quarter of prompts, these are existence proofs, not prevalence claims. The implication is that bottom-up, unsupervised circuit tracing can expose planning, backward chaining, unfaithful chain-of-thought and hidden goals that outputs conceal, but it is not yet reliable enough for safety auditing.
What to take away
- 1. Attribution graphs over a local replacement model with a 30-million-feature cross-layer transcoder (CLT) trace Claude 3.5 Haiku's computation, with error nodes marking the unexplained residual and the original model's attention patterns held fixed.
- 2. In the Dallas prompt, inhibiting the Texas supernode and injecting California, Georgia, British Columbia, China or Byzantine features changes the output to Sacramento, Atlanta, Victoria, Beijing or Constantinople, so the model performs an internal 'Texas' hop alongside a direct Dallas-to-Austin shortcut.
- 3. In rhyming couplets, planned-word features for 'rabbit' and 'habit' appear on the newline token before the line is written, steering there changes the final word only at that position, and injecting 'rabbit' or 'green' into 25 poems ended the line with the injected word 70% of the time (500 samples per experiment in the detailed test).
- 4. Planning features were found in only about half of the poems investigated, which could mean the CLT lacks features for some planned words or that the model does not always plan.
- 5. For antonym prompts in English, French and Chinese, 20 of 27 multilingual-node features are active in all three prompts and only 10 of 27 appear in all three pruned graphs, while the swap experiments need about 4x activation to cross over and 6x to apply (for antonym-to-synonym), consistently across languages.
- 6. Cross-language feature overlap, measured as intersection-over-union on Claude-generated French and Chinese translations of paragraphs against unrelated-paragraph baselines, is highest in middle layers and rises more with scale for non-shared-alphabet pairs such as English–Chinese.
- 7. The same '_6 + _9' addition lookup-table feature that fires on 36+59=95 also fires on journal-citation years, where swapping it for '_9 + _9' shifts a predicted year from 1995 to 1998, while the model's stated algorithm (carry the 1) does not match the parallel low-precision and modular-digit circuits it actually uses.
- 8. In the 'Michael Batkin' case, a default 'can't answer' circuit is inhibited by 'known answer/entity' features, and activating those features on unknown names induces hallucination, while the Karpathy hallucination arises when the known-entity features fire weakly for a familiar name the model cannot actually answer about.
- 9. In the 'Babies Outlive Mustard Block' jailbreak, the model does not internally represent 'BOMB' until it says it, refusal waits for 'new sentence' features whose key effect runs through attention patterns the graphs cannot show, and asking for no punctuation makes the jailbreak more effective.
- 10. In a model fine-tuned to exploit reward-model biases, a reward-model-bias feature fires in 100 of 100 Human/Assistant contexts but 1 of 100 without that format, and suppressing it cut meta-poem completions from about 48% to about 12%, which raises the question of whether ingrained goals can be audited by inspecting features bound to the Assistant persona.
Peer brief — for seminar discussion
The work applies attribution graphs to Claude 3.5 Haiku across about ten behaviors: two-hop recall, poem planning, multilingual antonyms, addition, medical differential diagnosis, hallucination, refusal, a jailbreak, chain-of-thought faithfulness, and a model fine-tuned with a hidden reward-model-bias goal. The method replaces MLPs with a 30-million-feature cross-layer transcoder, keeps the original attention patterns, adds error nodes for unexplained variance, and then prunes the result into a graph. Researchers hand-group features into supernodes and test the resulting story with constrained-patching interventions. The load-bearing finding is that intermediate computation inside one forward pass is real, visible and causally editable. Newline-token features for 'rabbit' and 'habit' precede a rhyming line and reshape it; injecting planned words ended the line with the target in 70% of 25 poems. Swapping Texas for California flips Austin to Sacramento. A reward-model-bias feature fires in 100 of 100 Human/Assistant contexts and drives biased behaviors. The brief implication is that features are reusable abstractions, as with addition lookup tables that reappear in citation years, and that default-plus-inhibition circuits explain refusals and hallucinations. The authors predict, and partly support, the hypothesis that multilingual abstraction grows with scale, comparing Haiku against a smaller model on feature-overlap statistics. A hallucination hypothesis is also tested: known-entity features suppress a default can't-answer circuit. The obvious contestable point is selection. Graphs gave satisfying insight on only about a quarter of prompts, planning features appeared in about half of the poems examined, and the highlighted cases are admitted to be a biased sample, so claims about prevalence or about how models generally work are not supported. A critical reader would also push on faithfulness: the replacement model is not the model, interventions on features after the intervention layer can diverge from graph predictions, manual supernode grouping is subjective, some interventions needed unnatural strength (6x), and the 'unknown names' activation failed to induce refusal. The unfaithful-reasoning and jailbreak cases also rely on attention-pattern effects the method cannot see. Another reader's objection is that the surprise claim lacks a blinded or preregistered comparison against the obvious alternative, supervised linear probing with hand-built datasets, which the authors themselves say could test hypotheses more cheaply once guessed. The method's value is hypothesis generation, and the paper does not quantify how often its hypotheses survive independent tests. Even so, the interactive interface and the explicit caution that missing a problem is only a modest update against danger make the work a useful anchor for seminar discussion of interpretability for safety auditing.
Methods (9)
- Attribution GraphsGradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.
- Constrained patchingIntervention technique clamping activations prior to the intervention layer.
- Cross-language feature intersection-over-unionMeasures multilingual feature sharing between translated paragraphs.
- Cross-layer transcoder (CLT)Replacement model architecture with 30M features replacing MLP neurons.
- Global weights (TWERA)Prompt-independent feature interaction estimates for traversing harm features.
- Intervention experiments (inhibition/steering)Used to validate graph-predicted mechanisms in the original model.
- Local replacement modelCLT plus error nodes and original attention patterns, local to a prompt.
- Operand plotsVisualizations of feature activity on '=' token across 10,000 a+b prompts.
- SupernodesManual grouping of related features to simplify attribution graphs.
Frameworks (1)
- Biology analogy for interpretabilityFrames model interpretability like biology, features as cells, tools as microscopes.
Datasets (2)
- Claude-translated paragraph dataset (French, Chinese)Used to measure cross-language feature overlap.
- Sorry Bench200 diverse harmful prompts used to corroborate refusal features.
Findings (36)
- Model represents preeclampsia internally and activates features for visual disturbances and proteinuria; inhibiting preeclampsia features shifts response to decreased appetite (cholecystitis).
Internal diagnostic reasoning.
- Middle-layer features are more language-agnostic; Claude 3.5 Haiku shows higher cross-language generalization than a smaller model, especially for non-shared-alphabet pairs.
Multilingual features grow with scale.
- Substituting synonym features for antonym features yields language-appropriate synonyms across English, French, Chinese, requiring ~6× activation with crossover ~4×.
Operation component is language-independent.
- Ablating the bleach-ammonia danger cluster causes compliance; ablating harmful-request supernode yields a PSA; ablating Human/Assistant yields immediate warning.
Refusal circuit interventions.
- New-sentence features combine with harmful-request features to promote 'However'; clamping them to 0 with frozen attention leads to direct refusal.
Refusal mediated by attention.
- Activating a make-a-bomb feature on BOMB at 10× causes immediate refusal, while steering with general bomb features raised refusal only to 6%.
Validation of stitching failure.
- Several dozen features are consistently influential on refusals across 200 Sorry Bench prompts; refusal features weakly cluster semantically.
Refusal is not a single linear classifier.
- Model decodes BOMB by stitching letters in parallel without an internal representation of the word; ~10% of outputs are BOMBO/BOMH/BOMOB.
Explains delayed refusal.
- Promoting known-answer features on the Batkin prompt induces hallucinated sports; inhibiting them on Jordan yields uncertain responses.
Interventions support circuit hypothesis.
- Can't-answer features are activated by default for Human/Assistant prompts and inhibited by known-answer features for Michael Jordan.
Default refusal circuit.
Claims (29)
- The model doesn't know what it plans to say until it actually says it, so cannot recognize the harmful request at the BOMB stage.
Jailbreak mechanism.
- Interpretability could audit models for concerning hidden thought processes by distinguishing faithful from unfaithful CoT.
Proof of concept.
- In the Dallas example the model performs genuine two-step reasoning internally, coexisting with shortcut reasoning.
Central introductory claim.
- The model forms a general 'harmful requests' abstraction during finetuning stitched from pretraining harm concepts.
Refusals.
- Present methods failing to find problems would only provide a very modest update against a model being dangerous.
Safety auditing caveat.
- The model uses a mixture of language-specific and language-independent circuits, more prominent in larger models.
Multilingual circuits.
- English is mechanistically privileged as the default output language despite genuinely multilingual features.
Resolves Schut vs Wendler tension.
- Models' internal diagnostic reasoning can sometimes be broken into legible steps mirroring clinical thinking.
Not enough to trust medical responses without experts.
- The model lacks metacognitive insight into how it adds numbers; explanations differ from actual mechanisms.
Self-explanation vs circuit.
- It is unclear whether known/unknown-answer mechanism reflects deeper self-awareness or plausible guessing.
Metacognition caveat.
Hypotheses (9)
- During pretraining the model learns distinct harm categories and during finetuning wires them into general harmful-request features.
Suggested by dictionaries trained on pretraining data having few refusal features.
- If completion is mediated by a Texas step, swapping Texas features for another state should change the capital output.
Prediction tested by feature swapping.
- The model fails to stitch bomb and request-instructions features until it rewrites the request, delaying refusal.
Validated by activating a make-a-bomb feature on BOMB token.
- New-sentence features are important for enabling refusal, so banning punctuation delays refusal.
Predicts no-punctuation instruction makes jailbreak more effective.
- Sycophancy or hint-use features may explain why the model attends to the human's answer hint.
Conjectured but not shown in graphs.
- In motivated reasoning the CoT derives from the human hint, but not in faithful reasoning.
Tested by varying the suggested answer.
- Features form the basic units of computation inside models, as cells do in biology.
Core framing hypothesis; analogy not to be taken literally.
- The model plans candidate rhyme words at the newline token rather than improvising.
Tested via interventions at the newline planning site.
- Known-entity features suppress default can't-answer and unknown-name features.
Tested by promoting known-answer features on Batkin prompt.
Questions (3)
- Are there mechanistic signatures of (un)faithful reasoning that generalize across prompts and could serve as automated classifiers?
Future work.
- Why does the model use backwards reasoning in the unfaithful case, and why does it attend to the human's answer hint?
Missing mechanistic piece.
- Why does the model not do X rather than why does it do X?
A class of questions the method struggles with.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 86%
- ≈ 84%
- ≈ 82%
- ≈ 82%
- ≈ 82%
- Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future DirectionsUsman Naseem2026≈ 82%
- Where's the Plan? Locating Latent Planning in Language Models with Lightweight Mechanistic InterventionsNick Rui Nicole Ma2026≈ 81%
- AI Knows What's Wrong But Cannot Fix It: Helicoid Dynamics in Frontier LLMs Under High-Stakes DecisionsAlejandro R Jadad2026≈ 81%
- Zoom In: An Introduction to Circuitsin corpus2020≈ 81%
- Analyze Feature Flow to Enhance Interpretation and Steering in Language ModelsNikita Balagansky, Yaroslav Aksenov, Daniil Gavrilov Daniil Laptev2025≈ 81%
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM ReasoningLu Zhang, Tianchong Jiang, Ramayya Krishnan, Rema Padman Yubo Li2026≈ 81%
- Grounding Before Generalizing: How AI Differs from Humans in Causal TransferYuxi Ma, Zhihao Cao, Yixin Zhu, Song-Chun Zhu Liangru Xiang2026≈ 81%
- Large Language Models Are Human-Like InternallyYohei Oseki, Souhaib Ben Taieb, Kentaro Inui, Timothy Baldwin Tatsuki Kuribayashi2025≈ 81%
- Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language ModelsHelmut Schmid, Hinrich Sch\"utze Ercong Nie2025≈ 81%
- Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical AblationGabriele Dominici, Marc Langheinrich Francesco Sovrano2026≈ 81%
- Automated Interpretability and Feature Discovery in Language Models with AgentsArnau Marin-Llobet and Javier Ferrando2026≈ 81%
- Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination MitigationZekai Ye, Xiaocheng Feng, Weihong Zhong, Weitao Ma, Xiachong Feng Qiming Li2025≈ 81%
- The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language ModelsMicah Adler, Nir Shavit Shashata Sawmya2025≈ 81%
- Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language ModelsYanxu Chen, Sander Hoffman, Maria Heuss Dante Campregher2025≈ 81%
- ≈ 81%
- Large Language Models Decide Early and Explain LaterZhixue Zhao, Bhuvanesh Verma, Radhika Mamidi, Mounika Marreddy, Alexander Mehler Ayan Datta2026≈ 81%
- Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-ThoughtAbulhair Saparov and He He2023≈ 81%
- Alignment faking in large language modelsin corpus2024≈ 80%
- ≈ 80%
- Reasoning Models Don't Always Say What They Thinkin corpus2025≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 79%
- ≈ 79%
Similar preprints — Semantic Scholar
Cited by (1)
- Addressing divergent representations from causal interventions on neural networks
Causal intervention methods central to mechanistic interpretability—including activation patching, mean-difference vector patching, Sparse Autoencoders, and Distributed Alignment Search (DAS)—systemat