paper:tracing-attention-8d61adTracing Attention Computation Through Feature Interactions
TL;DR
Attribution graphs built on transcoders froze attention patterns, so they could not say why a head attended where it did, and for many prompts that missing explanation is the whole computation. Tracing Attention fixes this with QK attributions, which write pre-softmax attention scores as a sum of query-feature by key-feature dot products through W_QK, plus head loadings that record each attention head's contribution to every graph edge. Head loadings require 'checkpointing' attention paths, because the number of head paths between features grows exponentially with layer separation. The paper settles on residual-stream SAEs, implemented as weakly causal crosscoders, over Multi-Token Transcoders or attention-output SAEs, which accumulate more reconstruction error. On Claude 3.5 Haiku the method confirms hypotheses that had been untestable. In the Aunt Sally induction prompt, query-side 'Aunt' features pair with key-side 'name of an Aunt/Uncle' features, and removing the key-side feature eliminates induction. In the French antonym prompt, query-side 'opposite' features find the tagged word 'petit'. In a multiple-choice prompt, 'correct answer' key features steer attention to option B. It also surfaces unexpected structure: concordance and discordance heads that judge whether a stated answer fits, without first computing it, on the banana-is-yellow and 18+5=23 prompts, and several parallel heuristics inside each circuit, such as 'attend to any name'. Attentional circuits are therefore mixtures of redundant mechanisms, and attribution graphs need QK information to be explanatory.
What to take away
- 1. QK attributions rewrite a head's pre-softmax attention score as a sum of bilinear dot products between query-position and key-position feature components, computed through W_QK = W_Q^T W_K, with bias and residual-error terms included.
- 2. Transcoder-based attribution graphs freeze attention patterns, so they ignore QK circuits entirely and leave ambiguous which head sequences mediate an edge.
- 3. Because the number of attention-head paths between transcoder features grows exponentially with their layer separation, the paper 'checkpoints' attention by using residual-stream SAEs (implemented as weakly causal crosscoders), which restricts every edge to paths of length 1.
- 4. The alternatives (Multi-Token Transcoders and attention-output SAEs) accumulate residual-stream error across layers and give attributions dominated by error nodes, but the chosen residual-stream approach costs linear MLP attributions and risks results that are uninterpretable or local to one prompt.
- 5. In the Haiku induction prompt 'I always loved visiting Aunt Sally… Aunt', heads with high loadings (roughly 3–10% of heads) combine query-side 'Aunt' features with key-side 'name' and 'name of an Aunt/Uncle' features, and scaling the latter down on the key side removes induction so the model predicts generic aunt names.
- 6. In the French antonym prompt 'Le contraire de "petit" est "', inhibiting query-side 'opposite' features makes Haiku predict synonyms such as 'peu' and 'faible' instead of 'grand', and inhibiting key-side 'adjective' features has a similar but smaller effect.
- 7. In the World War II multiple-choice prompt, steering 'correct answer' features on key-side option tokens flips Haiku's answer from B to another letter, which confirms a mechanism where heads attend to the correct option with a query-side bias toward correct answers.
- 8. On 'The color of a banana is yellow?' and '18+5=23?', Haiku shows no clear evidence of first computing the answer, because separate concordance and discordance heads detect query-key feature agreement, and steering 'red' up and 'yellow' down on the query side flips the answer from Yes to No.
- 9. Flipping 'No' to 'Yes' required larger steering magnitudes on concordance heads than flipping 'Yes' to 'No' on discordance heads, which suggests that 'more things have to go right' for an answer to be judged correct, a hypothesis the paper leaves untested.
- 10. A replicable control: steering many heads was needed to move the induction behavior, and freezing the attention patterns of non-steered heads cut the required number to under half, consistent with a 'hydra head' compensation effect.
Peer brief — for seminar discussion
Frozen attention patterns were a known hole in transcoder attribution graphs. The graphs said a 'Sally' feature caused the 'Sally' logit but never said why the model attended to that token. This paper adds two instruments: QK attributions, which decompose pre-softmax scores into query-feature by key-feature interaction terms via W_QK, and head loadings, which assign each graph edge's strength to individual attention heads. Loadings need checkpointing, since head paths between transcoder features grow exponentially with layer gap. The authors use residual-stream SAEs, in practice weakly causal crosscoders, rather than Multi-Token Transcoders or attention-output SAEs, which accumulated error. A path-search algorithm over transcoder graphs would have been an alternative. The case studies use Claude 3.5 Haiku. Induction on 'Aunt Sally' arises from query 'Aunt' features meeting key 'name of an Aunt/Uncle' features. Antonym lookup for 'petit' and multiple-choice answering for the 1945 option both work by tagging a token that later queries find. The 'banana is yellow' and '18+5=23' prompts show concordance and discordance heads that check a stated answer without computing the true one. Steering a handful of features flips outputs, for example Yes to No, which supports the circuits causally. The mechanisms are also redundant, with a general 'attend to any name' or 'any answer' heuristic running alongside the core one, and the induction intervention needed roughly 3–10% of heads, which suggests hydra-style compensation. The implication is that attribution graphs without QK information often miss the crux, and that attentional circuits are mixtures of parallel heuristics rather than single clean mechanisms. The paper hypothesizes that checking correctness demands that 'more things go right', based on the asymmetric steering magnitudes. A critical reader would push back on evidence strength. Every case is one hand-picked prompt on one model, with head labels assigned from that prompt alone, and the authors concede the heads may be polysemantic elsewhere. Interventions show necessity and sufficiency for the output, but they do not show the decomposition is unique or complete, since error nodes obscured the multiple-choice 'correct answer' computation. The softmax competition problem is also unsolved: inhibitory terms need manual inspection. The chosen SAE route also gives up linear MLP attribution, which risks prompt-local explanations. Quantitative comparison against patching baselines is absent, as is any measure of how much attention-score variance the top terms explain. The 'more things have to go right' account rests on a single steering asymmetry.
Methods (11)
- Attribution GraphsGradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.
- Checkpointing Attention Paths with FeaturesStrategy forcing edges to be mediated only by head paths of length 1 to avoid exponential path growth.
- Edge attribution patchingUsed by prior work to identify consistency heads.
- Feature steering interventions within QK circuitsScaling features inside QK circuits of selected heads to validate mechanisms.
- Head LoadingsPer-edge contribution of each attention head to attribution graph edges.
- QK AttributionsCore method decomposing attention scores as bilinear sums of query-key feature pairs.
- Residual stream SAEs with adjacent-layer gradient attributionsThird checkpointing strategy; adopted here.
- SAEs on attention layer outputsSecond checkpointing strategy.
- sparse autoencodersExisting method for model interpretability that decodes model activations rather than parameters themselves, noted as incomplete solution.
- TranscodersDecomposition method for activations; VPD is compared against transcoders in sparsity-reconstruction tradeoff.
- Weakly causal crosscoder (10M features)Source of the features used in attributions.
Frameworks (3)
- Cross-Layer TranscodersMLP replacement with sparse features underlying the original attribution graphs.
- Multi-token transcodersDecomposition approach mixing frozen base attention patterns.
- Sparse CrosscodersExtension of SAEs that jointly learns latents across representations from different models; proposed as alternative for extended fine-tuning
Findings (19)
- Small-to-large edges in the French antonym prompt are mediated by a limited set of heads using key-side opposite-requested/adjective features and query-side antonym features
QK attribution of antonym heads.
- Inhibiting query-side opposite features reduces 'grand' prediction and induces synonyms like peu and faible; inhibiting key-side adjective features has lesser effect
Steering validation for antonym mechanism.
- Multiple-choice heads attend via query-side answer-needed features and bias to key-side correct-answer features, inhibited by false-statement features
QK attribution for MC heads.
- Same concordance and discordance heads discriminate correct from incorrect on 18+5=23 vs 18+6=23, driven by 'numbers separated by 5' query features
Generalization to arithmetic.
- No clear evidence that Haiku first computes color+banana=yellow before matching the stated answer; separate heads detect concordance/discordance
Contradicts initial hypothesis.
- In the Aunt Sally prompt, query-side Aunt features interact with key-side name and aunt/uncle-name features in Haiku induction heads
QK attribution of induction-relevant heads.
- Scaling the output of a single concordance or discordance head suffices to flip the response; effect strengthens with multiple heads
Head-level causal evidence.
- Steering red up and yellow down on query side flips Haiku's answer Yes to No by shifting concordance and discordance head attention
Causal validation.
- Removing key-side 'name of aunt/uncle' features in QK circuits eliminates induction in Haiku, which predicts generic aunt names
Steering validation.
- Steering induction required 3–10% of heads, and fewer than half as many when non-steered heads' attention patterns were frozen
Evidence for hydra effect.
Claims (11)
- Attention scores are a bilinear function of residual stream at query and key positions, so can be written as sums of feature-feature dot products
Core insight.
- QK attributions are a significant qualitative improvement on original attribution graphs, unlocking previously impossible analyses
Central claim.
- More things have to go right for a stated answer to be deemed correct; discordance heads signal strong incorrectness
Interpretation of steering asymmetry.
- The model mixes a mechanism tagging 'petit' as the word to invert with one searching for any plausible adjective
Antonym interpretation.
- Concordance/discordance heads use QK circuits to check for concordant properties between query and key features
Preliminary conclusion.
- Multiple-choice heads have a default bias to attend to correct answers, strengthened when an answer is needed
MC interpretation.
- Transcoder-based attribution graphs omit attentional computation, rendering them useless for many prompts
Motivating problem.
- Number of attention head paths between transcoder features grows exponentially with layer separation
Why checkpointing is needed.
- Attentional circuits employ many computations and heuristics in parallel, even in simple contexts
New mechanism insight.
- Induction QK circuits combine searching for any name and searching for names of aunts/uncles
Interpretation of induction heads.
Hypotheses (4)
- Multiple choice answers are found by copying B to 1945, marking it correct, attending via query interaction, and copying B back
Prior hypothesis now confirmed.
- Small-to-large transformation is mediated by opposite heads that invert semantic meaning
Prior hypothesis not confirmed.
- The model first computes color+banana=yellow and then matches the stated answer
Initial hypothesis not supported.
- Many heads must be steered because of the hydra head effect
Explanation for steering counts.
Questions (5)
- Are heads mediating the same edge attending for the same reason, suggesting attention head superposition?
Distributed functional units.
- Will a head perform 'the same algorithm' across prompts, or are individual heads polysemantic?
Head polysemanticity.
- How do we identify the most important negative contributions in an automated fashion?
Softmax competition problem.
- How can we scale head loadings and QK attributions to long contexts?
Quadratic scaling.
- Can QK attribution information be distilled or simplified?
Future work.
Original abstract (expand)
This paper describes a method to explain attention patterns in transformer models by decomposing them into feature interactions and integrating this information into attribution graphs. The authors introduce "QK attributions" to describe attention head scores as bilinear functions of feature activations at query and key positions, enabling mechanistic explanations of why attention heads attend to particular context positions. Through case studies in induction prompts, word opposites, and multiple choice questions, they demonstrate how this method unlocks previously impossible analyses of attentional circuits and reveals both expected and unexpected computational mechanisms.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Short-circuiting Shortcuts: Mechanistic Investigation of Shortcuts in Text ClassificationShihan Wang, Antske Fokkens Leon Eshuijs2025≈ 85%
- ≈ 85%
- Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language ModelsYanxu Chen, Sander Hoffman, Maria Heuss Dante Campregher2025≈ 85%
- Finding Interpretable Prompt-Specific Circuits in Language ModelsLucas M. Tassis, Azalea Rohr, Mark Crovella Gabriel Franco2026≈ 84%
- ≈ 84%
- Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination MitigationZekai Ye, Xiaocheng Feng, Weihong Zhong, Weitao Ma, Xiachong Feng Qiming Li2025≈ 84%
- ≈ 83%
- Automated Interpretability and Feature Discovery in Language Models with AgentsArnau Marin-Llobet and Javier Ferrando2026≈ 83%
- ≈ 83%
- A Mathematical Framework for Transformer Circuitsin corpus2021≈ 83%
- Analyze Feature Flow to Enhance Interpretation and Steering in Language ModelsNikita Balagansky, Yaroslav Aksenov, Daniil Gavrilov Daniil Laptev2025≈ 83%
- From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder FeaturesPablo Magari\~nos-Docampo, Javier Perez-Robles Ruben Fernandez-Boullon2026≈ 83%
- Causal Probing for Internal Visual Representations in Multimodal Large Language ModelsTianjie Ju, Zheng Wu, Liangbo He, Jun Lan, Huijia Zhu, Weiqiang Wang, Zhuosheng Zhang Zehao Deng2026≈ 83%
- Reasoning Resides in Layers: Restoring Temporal Reasoning in Video-Language Models with Layer-Selective MergingHaonan Wang, Jian Kang, Kenji Kawaguchi, Jiaying Wu Zihang Fu2026≈ 82%
- Stream: Scaling up Mechanistic Interpretability to Long Context in LLMs via Sparse AttentionJos\'e Luis Redondo Garc\'ia, Gustavo Penha, Konstantina Palla, Hugues Bouchard J Rosser2026≈ 82%
- ≈ 82%
- Mechanistic Exploration of Backdoored Large Language Model Attention PatternsLakshmi Babu-Saheer Mohammed Abu Baker2025≈ 82%
- ≈ 82%
- ≈ 82%
- Extracting Rule-based Descriptions of Attention Features in TransformersAdithya Bhaskar, Alexander Wettig, Danqi Chen Dan Friedman2025≈ 82%
- ≈ 81%
- ≈ 81%
- ≈ 81%
- ≈ 81%
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalin corpus2026≈ 80%
- Reasoning Models Don't Always Say What They Thinkin corpus2025≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 80%
Similar preprints — Semantic Scholar
Cited by (1)
- When Models Manipulate Manifolds: The Geometry of a Counting Task
Claude 3.5 Haiku's linebreaking in fixed-width text is implemented by manipulating low-dimensional, rippled 1-D feature manifolds: character count, line width, characters remaining and next-token leng