paper
active
paper:tracing-attention-8d61ad

Tracing Attention Computation Through Feature Interactions

TL;DR

Attribution graphs built on transcoders froze attention patterns, so they could not say why a head attended where it did, and for many prompts that missing explanation is the whole computation. Tracing Attention fixes this with QK attributions, which write pre-softmax attention scores as a sum of query-feature by key-feature dot products through W_QK, plus head loadings that record each attention head's contribution to every graph edge. Head loadings require 'checkpointing' attention paths, because the number of head paths between features grows exponentially with layer separation. The paper settles on residual-stream SAEs, implemented as weakly causal crosscoders, over Multi-Token Transcoders or attention-output SAEs, which accumulate more reconstruction error. On Claude 3.5 Haiku the method confirms hypotheses that had been untestable. In the Aunt Sally induction prompt, query-side 'Aunt' features pair with key-side 'name of an Aunt/Uncle' features, and removing the key-side feature eliminates induction. In the French antonym prompt, query-side 'opposite' features find the tagged word 'petit'. In a multiple-choice prompt, 'correct answer' key features steer attention to option B. It also surfaces unexpected structure: concordance and discordance heads that judge whether a stated answer fits, without first computing it, on the banana-is-yellow and 18+5=23 prompts, and several parallel heuristics inside each circuit, such as 'attend to any name'. Attentional circuits are therefore mixtures of redundant mechanisms, and attribution graphs need QK information to be explanatory.

What to take away

  1. 1. QK attributions rewrite a head's pre-softmax attention score as a sum of bilinear dot products between query-position and key-position feature components, computed through W_QK = W_Q^T W_K, with bias and residual-error terms included.
  2. 2. Transcoder-based attribution graphs freeze attention patterns, so they ignore QK circuits entirely and leave ambiguous which head sequences mediate an edge.
  3. 3. Because the number of attention-head paths between transcoder features grows exponentially with their layer separation, the paper 'checkpoints' attention by using residual-stream SAEs (implemented as weakly causal crosscoders), which restricts every edge to paths of length 1.
  4. 4. The alternatives (Multi-Token Transcoders and attention-output SAEs) accumulate residual-stream error across layers and give attributions dominated by error nodes, but the chosen residual-stream approach costs linear MLP attributions and risks results that are uninterpretable or local to one prompt.
  5. 5. In the Haiku induction prompt 'I always loved visiting Aunt Sally… Aunt', heads with high loadings (roughly 3–10% of heads) combine query-side 'Aunt' features with key-side 'name' and 'name of an Aunt/Uncle' features, and scaling the latter down on the key side removes induction so the model predicts generic aunt names.
  6. 6. In the French antonym prompt 'Le contraire de "petit" est "', inhibiting query-side 'opposite' features makes Haiku predict synonyms such as 'peu' and 'faible' instead of 'grand', and inhibiting key-side 'adjective' features has a similar but smaller effect.
  7. 7. In the World War II multiple-choice prompt, steering 'correct answer' features on key-side option tokens flips Haiku's answer from B to another letter, which confirms a mechanism where heads attend to the correct option with a query-side bias toward correct answers.
  8. 8. On 'The color of a banana is yellow?' and '18+5=23?', Haiku shows no clear evidence of first computing the answer, because separate concordance and discordance heads detect query-key feature agreement, and steering 'red' up and 'yellow' down on the query side flips the answer from Yes to No.
  9. 9. Flipping 'No' to 'Yes' required larger steering magnitudes on concordance heads than flipping 'Yes' to 'No' on discordance heads, which suggests that 'more things have to go right' for an answer to be judged correct, a hypothesis the paper leaves untested.
  10. 10. A replicable control: steering many heads was needed to move the induction behavior, and freezing the attention patterns of non-steered heads cut the required number to under half, consistent with a 'hydra head' compensation effect.

Peer brief — for seminar discussion

Frozen attention patterns were a known hole in transcoder attribution graphs. The graphs said a 'Sally' feature caused the 'Sally' logit but never said why the model attended to that token. This paper adds two instruments: QK attributions, which decompose pre-softmax scores into query-feature by key-feature interaction terms via W_QK, and head loadings, which assign each graph edge's strength to individual attention heads. Loadings need checkpointing, since head paths between transcoder features grow exponentially with layer gap. The authors use residual-stream SAEs, in practice weakly causal crosscoders, rather than Multi-Token Transcoders or attention-output SAEs, which accumulated error. A path-search algorithm over transcoder graphs would have been an alternative. The case studies use Claude 3.5 Haiku. Induction on 'Aunt Sally' arises from query 'Aunt' features meeting key 'name of an Aunt/Uncle' features. Antonym lookup for 'petit' and multiple-choice answering for the 1945 option both work by tagging a token that later queries find. The 'banana is yellow' and '18+5=23' prompts show concordance and discordance heads that check a stated answer without computing the true one. Steering a handful of features flips outputs, for example Yes to No, which supports the circuits causally. The mechanisms are also redundant, with a general 'attend to any name' or 'any answer' heuristic running alongside the core one, and the induction intervention needed roughly 3–10% of heads, which suggests hydra-style compensation. The implication is that attribution graphs without QK information often miss the crux, and that attentional circuits are mixtures of parallel heuristics rather than single clean mechanisms. The paper hypothesizes that checking correctness demands that 'more things go right', based on the asymmetric steering magnitudes. A critical reader would push back on evidence strength. Every case is one hand-picked prompt on one model, with head labels assigned from that prompt alone, and the authors concede the heads may be polysemantic elsewhere. Interventions show necessity and sufficiency for the output, but they do not show the decomposition is unique or complete, since error nodes obscured the multiple-choice 'correct answer' computation. The softmax competition problem is also unsolved: inhibitory terms need manual inspection. The chosen SAE route also gives up linear MLP attribution, which risks prompt-local explanations. Quantitative comparison against patching baselines is absent, as is any measure of how much attention-score variance the top terms explain. The 'more things have to go right' account rests on a single steering asymmetry.

Methods (11)

Frameworks (3)

  • Cross-Layer Transcoders
    MLP replacement with sparse features underlying the original attribution graphs.
  • Multi-token transcoders
    Decomposition approach mixing frozen base attention patterns.
  • Sparse Crosscoders
    Extension of SAEs that jointly learns latents across representations from different models; proposed as alternative for extended fine-tuning

Findings (19)

Original abstract (expand)

This paper describes a method to explain attention patterns in transformer models by decomposing them into feature interactions and integrating this information into attribution graphs. The authors introduce "QK attributions" to describe attention head scores as bilinear functions of feature activations at query and key positions, enabling mechanistic explanations of why attention heads attend to particular context positions. Through case studies in induction prompts, word opposites, and multiple choice questions, they demonstrate how this method unlocks previously impossible analyses of attentional circuits and reveals both expected and unexpected computational mechanisms.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar

Cited by (1)