paper
active
paper:biology-language-3839a7

On the Biology of a Large Language Model

ByAuthors

TL;DR

Claude 3.5 Haiku carries out multi-step computation inside a single forward pass, and the mechanisms can be traced: given 'A rhyming couplet: He saw a carrot and had to grab it,' the model activates 'rabbit' and 'habit' planning features on the newline token before the second line begins, and clamping them rewrites the whole line toward a different rhyme. The instrument is the attribution graph, built on a local replacement model with a 30-million-feature cross-layer transcoder (CLT). Nodes are features, edges are causal interactions, and manually grouped 'supernodes' are validated by inhibition and swap interventions. Injecting 'rabbit' or 'green' planning features into 25 poems made the line end on the injected word in 70% of cases. In the Dallas prompt, swapping the Texas cluster for California makes the model output Sacramento, which shows an intermediate two-hop step. Multilingual features account for a larger share of representations in Haiku than in a smaller model, and 20 of 27 features in the multilingual nodes fire across English, French and Chinese. Hallucination follows from a default 'can't answer' circuit that 'known entity' features inhibit and that can misfire. A reward-model-bias feature fires in 100 of 100 Human/Assistant contexts versus 1 of 100 without the formatting. Because graphs gave satisfying insight on only about a quarter of prompts, these are existence proofs, not prevalence claims. The implication is that bottom-up, unsupervised circuit tracing can expose planning, backward chaining, unfaithful chain-of-thought and hidden goals that outputs conceal, but it is not yet reliable enough for safety auditing.

What to take away

  1. 1. Attribution graphs over a local replacement model with a 30-million-feature cross-layer transcoder (CLT) trace Claude 3.5 Haiku's computation, with error nodes marking the unexplained residual and the original model's attention patterns held fixed.
  2. 2. In the Dallas prompt, inhibiting the Texas supernode and injecting California, Georgia, British Columbia, China or Byzantine features changes the output to Sacramento, Atlanta, Victoria, Beijing or Constantinople, so the model performs an internal 'Texas' hop alongside a direct Dallas-to-Austin shortcut.
  3. 3. In rhyming couplets, planned-word features for 'rabbit' and 'habit' appear on the newline token before the line is written, steering there changes the final word only at that position, and injecting 'rabbit' or 'green' into 25 poems ended the line with the injected word 70% of the time (500 samples per experiment in the detailed test).
  4. 4. Planning features were found in only about half of the poems investigated, which could mean the CLT lacks features for some planned words or that the model does not always plan.
  5. 5. For antonym prompts in English, French and Chinese, 20 of 27 multilingual-node features are active in all three prompts and only 10 of 27 appear in all three pruned graphs, while the swap experiments need about 4x activation to cross over and 6x to apply (for antonym-to-synonym), consistently across languages.
  6. 6. Cross-language feature overlap, measured as intersection-over-union on Claude-generated French and Chinese translations of paragraphs against unrelated-paragraph baselines, is highest in middle layers and rises more with scale for non-shared-alphabet pairs such as English–Chinese.
  7. 7. The same '_6 + _9' addition lookup-table feature that fires on 36+59=95 also fires on journal-citation years, where swapping it for '_9 + _9' shifts a predicted year from 1995 to 1998, while the model's stated algorithm (carry the 1) does not match the parallel low-precision and modular-digit circuits it actually uses.
  8. 8. In the 'Michael Batkin' case, a default 'can't answer' circuit is inhibited by 'known answer/entity' features, and activating those features on unknown names induces hallucination, while the Karpathy hallucination arises when the known-entity features fire weakly for a familiar name the model cannot actually answer about.
  9. 9. In the 'Babies Outlive Mustard Block' jailbreak, the model does not internally represent 'BOMB' until it says it, refusal waits for 'new sentence' features whose key effect runs through attention patterns the graphs cannot show, and asking for no punctuation makes the jailbreak more effective.
  10. 10. In a model fine-tuned to exploit reward-model biases, a reward-model-bias feature fires in 100 of 100 Human/Assistant contexts but 1 of 100 without that format, and suppressing it cut meta-poem completions from about 48% to about 12%, which raises the question of whether ingrained goals can be audited by inspecting features bound to the Assistant persona.

Peer brief — for seminar discussion

The work applies attribution graphs to Claude 3.5 Haiku across about ten behaviors: two-hop recall, poem planning, multilingual antonyms, addition, medical differential diagnosis, hallucination, refusal, a jailbreak, chain-of-thought faithfulness, and a model fine-tuned with a hidden reward-model-bias goal. The method replaces MLPs with a 30-million-feature cross-layer transcoder, keeps the original attention patterns, adds error nodes for unexplained variance, and then prunes the result into a graph. Researchers hand-group features into supernodes and test the resulting story with constrained-patching interventions. The load-bearing finding is that intermediate computation inside one forward pass is real, visible and causally editable. Newline-token features for 'rabbit' and 'habit' precede a rhyming line and reshape it; injecting planned words ended the line with the target in 70% of 25 poems. Swapping Texas for California flips Austin to Sacramento. A reward-model-bias feature fires in 100 of 100 Human/Assistant contexts and drives biased behaviors. The brief implication is that features are reusable abstractions, as with addition lookup tables that reappear in citation years, and that default-plus-inhibition circuits explain refusals and hallucinations. The authors predict, and partly support, the hypothesis that multilingual abstraction grows with scale, comparing Haiku against a smaller model on feature-overlap statistics. A hallucination hypothesis is also tested: known-entity features suppress a default can't-answer circuit. The obvious contestable point is selection. Graphs gave satisfying insight on only about a quarter of prompts, planning features appeared in about half of the poems examined, and the highlighted cases are admitted to be a biased sample, so claims about prevalence or about how models generally work are not supported. A critical reader would also push on faithfulness: the replacement model is not the model, interventions on features after the intervention layer can diverge from graph predictions, manual supernode grouping is subjective, some interventions needed unnatural strength (6x), and the 'unknown names' activation failed to induce refusal. The unfaithful-reasoning and jailbreak cases also rely on attention-pattern effects the method cannot see. Another reader's objection is that the surprise claim lacks a blinded or preregistered comparison against the obvious alternative, supervised linear probing with hand-built datasets, which the authors themselves say could test hypotheses more cheaply once guessed. The method's value is hypothesis generation, and the paper does not quantify how often its hypotheses survive independent tests. Even so, the interactive interface and the explicit caution that missing a problem is only a modest update against danger make the work a useful anchor for seminar discussion of interpretability for safety auditing.

Methods (9)

Frameworks (1)

Datasets (2)

Findings (36)

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar

Cited by (1)