paper:headvis-interactive-90fa42HeadVis: An Interactive Tool For Investigating Attention Heads
TL;DR
A single attention head's behavior on the full data distribution rarely matches what a narrow task suggests, and HeadVis, an interactive tool for browsing heads in Claude Haiku 3.5 and Gemma 3 1B, makes that gap visible. HeadVis combines a metric scatter plot of heads, attention patterns bucketed by max attention, PCA of Q/K/O activations, and QK/OV attributions over a 10M-feature weakly causal crosscoder. Four case studies show how far the workflow reaches. A fuzzy poetry induction head matches on 'ee' sounds in QK and copies literally in OV. An early-layer line width head turns out to be polysemantic: it handles years 1000–1986, multi-token words and newlines, and PCA separates all three behaviors. The answer selection head reuses one selection template well beyond multiple choice, but its PCA is a single cloud and the authors only guess it is monosemantic. A same-set suppression head sends a country's attention to the one foreign city in 98% of 185 sentences, with mean query-key cosine −0.47 against its own cities versus +0.01 elsewhere. Even so, it fits real dataset examples poorly. Heads can be polysemantic without superposition, since their outputs are high rank. This gives four obstacles to an attention analogue of transcoders: unknown attention biology, head polysemanticity, attention superposition and high-rank attentional features. The paper argues that these crisp, findable examples now give decomposition methods concrete targets.
What to take away
- 1. HeadVis pairs a configurable per-head metric scatter plot, attention patterns bucketed by max attention, PCA of Q/K/O activations, and real-time QK/OV feature attributions through a 10M-feature weakly causal crosscoder.
- 2. A fuzzy poetry induction head matches 'ee'-sound line endings in QK (a '-ea sound at line end' query feature against a 'newline after an ee sound' key feature) while its OV circuit copies exactly like literal induction, so the fuzziness lives entirely in QK.
- 3. An early-layer line width head in Claude Haiku 3.5 is polysemantic: it attends between the tokens of years such as 1776 (split into 1 + 776), between pieces of multi-token words, and between newlines, and PCA of Q, K and O separates all three behaviors into clusters.
- 4. Sweeping the year in a fixed prompt shows the line width head attends strongly only for 1000–1986, a constraint that neither the attention patterns nor the per-example attributions revealed.
- 5. The answer selection head's PCA is one continuous cloud with no clusters, so the line width head's PCA check cannot diagnose its polysemanticity.
- 6. Projecting the 'correct answer' feature through the answer selection head's key and query weights lands on other 'this is the relevant content' features that are near-orthogonal in the residual stream, whereas the line width head's years feature lands only near other year features. This supports a single general selection behavior.
- 7. In 185 sentences of the form '{five cities} are all cities in {country}', the same-set suppression head attends more to the one foreign city than to any of the four native cities 98% of the time.
- 8. Mean-subtracted country query vectors have mean cosine −0.47 with their own cities' key vectors and +0.01 with other cities, even though the residual-stream cosine is +0.17, so W_Q and W_K flip the sign.
- 9. Replacing 'Ivory Coast' with 'Zambia' in a sequence about a Zambian nonprofit dropped attention to the token 'a' from 0.87 to 0.08, and the median across the other 51 African countries was 0.27. An earlier head's K-composition writes the Zambian features onto 'a'.
- 10. Open question: can an attention layer be decomposed into sparse 'attentional features' that are possibly high rank, with the line width head giving three separate features and the answer selection head one high-rank feature, and is attention superposition real in any LLM? No confirmed example exists yet.
Peer brief — for seminar discussion
HeadVis is an open-source interactive tool, the attention-head analogue of feature visualization, for studying heads over a full data distribution rather than a single task. It combines a scatter plot of per-head metrics such as an induction score, top-activating attention patterns, PCA of Q/K/O activations, and real-time QK/OV attributions through a 10M-feature weakly causal crosscoder. The paper applies it to heads in Claude Haiku 3.5 and releases a demo on Gemma 3 1B. Four case studies run from easy to hard. A fuzzy induction head matches in feature space (an 'ee' rhyme) in QK and copies literally in OV. A line width head turns out to be cleanly polysemantic, with year, multi-token-word and newline behaviors that separate in PCA, and its year behavior only fires for 1000–1986. The answer selection head appears to reuse one 'relevant content' selection template across multiple choice, sentiment and dialogue, although PCA shows no clusters. A same-set suppression head sends a country's attention to the one foreign city in 98% of 185 synthetic sentences, with a −0.47 mean QK cosine against its own cities. The load-bearing finding is the pattern across the four: narrow-task behavior underdetermines what a head does, and the hard cases need feature-level tools because token patterns mislead. In the Ivory Coast example, attention depends on a Zambian feature that an earlier head wrote onto an unrelated token. The implication is that attention needs its own decomposition, and the discussion names four obstacles: attention biology, head polysemanticity without superposition, attention superposition, and high-rank features. These are compared with existing rank-1 OV approaches such as attention-output SAEs and Lorsa. The paper predicts that a good decomposition would split the line width head into three features and keep the answer selection head as one. A critical reader would push back on evidence strength and on how representative the heads are. The monosemanticity claim for answer selection rests on a handful of examples and virtual-weight neighbors, with no text hypothesis tested against a grader. The suppression mechanism was shown mainly on synthetic prompts and fits real data loosely. The heads were also chosen because they were interpretable, so the generalization from them is unclear. PCA, which the authors call hypothesis generation, can look clean or fail for reasons unrelated to how many behaviors a head has. A more direct alternative would be to split the head into separate heads and check that they reproduce its output. Another would be ablation or causal patching, but the paper uses neither.
Methods (13)
- 185-sentence country-city testSystematic test of same-set suppression with five cities, one foreign.
- Custom-sequence inputServer feature letting users type sequences and view head attention; not open-sourced.
- Feature VisualizationMethod of optimizing input to cause a neuron to fire maximally, used to characterize what a neuron detects; establishes causal link
- Head ranking on a (query, key) pairRanks all heads by attention on a pair to trace composition.
- HeadVis head selector scatter plotConfigurable scatter plot of heads by metrics to find interesting heads.
- HeadVis PCA view of Q, K, O activationsHypothesis-generation tool for detecting distinct head behaviors via clustering.
- Induction scoreProxy metric: average attention on literal-copy induction token pairs on a standard dataset.
- Mean-subtracted query/key cosine similarity analysisMeasures QK geometry between country queries and city keys.
- Prefix scorePrior metric on synthetic repeated random strings; induction score analogous.
- QK and OV attributionsWrites attention scores/head output as sums of feature interactions; core HeadVis capability.
- TranscodersDecomposition method for activations; VPD is compared against transcoders in sparsity-reconstruction tradeoff.
- Virtual weights between featuresUsed to project features through QK weights and find aligned features.
- Weakly causal crosscoder (10M features)Source of the features used in attributions.
Frameworks (5)
- Attention-output SAEsDecomposition approach with rank-1 OV directions.
- Attentional featuresProposed attention analog of sparse monosemantic features for decomposing attention layers.
- IOI circuitPrior circuit containing an inhibition head.
- Low-Rank Sparse Attention (Lorsa)Learns new full-rank QK circuits shared across feature groups.
- Multi-token transcodersDecomposition approach mixing frozen base attention patterns.
Datasets (2)
- Claude Haiku 3.5 attention headsModel whose heads are studied in the case studies.
- Gemma 3 1BOpen model used in the HeadVis demo.
Findings (21)
- Answer selection OV circuit always copies attended content while QK matches 'content marked relevant' to 'about to produce matching content'
Across MCQ, sentiment, dialogue.
- Features aligned with 'correct answer' through key/query weights are other 'relevant content' features, near-orthogonal in residual stream
Supports monosemantic reading.
- Fuzzy poetry induction head QK matches '-ea sound at line end' query with 'newline after ee sound' key feature
Fuzziness lives in QK.
- An earlier head attends from 'a' to chikumbuso and writes the Zambian features read by the suppression head
K-composition example.
- Fuzzy poetry induction head OV circuit is identical to literal induction (newline value to 'say \n' output)
Copying behavior unchanged.
- Country token attends more to the foreign city than to any own city 98% of the time across 185 sentences
Systematic test of same-set suppression.
- Fuzzy induction heads match words across languages, antonyms, structural positions and kinship terms
Variety of fuzzy matching found.
- In the residual stream a country and its own cities have positive cosine similarity (mean +0.17)
Head flips sign via W_Q and W_K.
- QK attribution from Ivory to 'a' has top pair Ivory Coast query feature and Zambian key feature
Explains attention edge.
- None of the top-activating examples for the answer selection head are multiple choice questions
Shows broader behavior.
Claims (15)
- Answer selection is a more general behavior: repeating previous context by identifying the entity about to be mentioned
Generalizes beyond multiple choice.
- An attention head can be polysemantic without superposition because its output is high-rank, unlike an MLP neuron
Theoretical point.
- Rank-1 OV decompositions split higher-rank OV behaviors like succession or copying induction across many features
Limitation of existing decompositions.
- What was isolated is part of what the head does with places, not cleanly separable nor a full account
Author's caveat.
- The field is positioned to tackle attention obstacles directly now that crisp examples exist
Conclusion.
- The fuzziness of fuzzy induction lives entirely in QK; OV copies like literal induction
Interpretation of attributions.
- A head's behavior on the full distribution rarely matches what a narrow task suggests
Theme across prior and own work.
- Sometimes what matters at a position is the features active there, not the token
Explains hard-to-read dataset patterns.
- Fuzzy induction can be understood as performing induction in the feature basis
Central interpretation of induction case study.
- The line width head is polysemantic, implementing several unrelated behaviors
Author is fairly confident.
Hypotheses (3)
- With an attention decomposition, the line width head's three behaviors would appear as three separate features and answer selection as one high-rank feature
Prediction for future decomposition.
- Some attentional features are spread across multiple heads
Suspected but no confirmed example in real LLM.
- The answer selection head is monosemantic
Best guess but not fully convinced; lacks crisp text description to test.
Questions (2)
- Does the answer selection head really implement one behavior, or many behaviors that share a common template?
Raised by disjoint feature sets.
- Is softmax competition for priority logic a fundamental part of the attention mechanism?
Open question about competition among keys.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Cross-Attention Head Position Patterns Can Align with Human Visual Concepts in Text-to-Image Generative ModelsJungmin Ko, Dongnam Byun, Jangwon Suh, Wonjong Rhee Jungwon Park2025≈ 81%
- Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable PersonalizationYe Yuan, Changjiang Han, Yuxing Tian, Zipeng Sun, Linfeng Du, Jikun Kang, Hong Kang, Xue Liu, Haolun Wu Weixu Zhang2026≈ 79%
- Successor Heads: Recurring, Interpretable Attention Heads In The WildEuan Ong, George Ogden, Arthur Conmy Rhys Gould2023≈ 79%
- ≈ 79%
- What Does BERT Look At? An Analysis of BERT's AttentionUrvashi Khandelwal, Omer Levy, Christopher D. Manning Kevin Clark2019≈ 78%
- Locating Demographic Bias at the Attention-Head Level in CLIP's Vision EncoderKittipat Phunjanna, Marcos Escudero Vi\~nolo, Catarina Barata, Jenny Benois-Pineau Alaa Yasser2026≈ 78%
- A Mathematical Framework for Transformer Circuitsin corpus2021≈ 78%
- Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language ModelsYanxu Chen, Sander Hoffman, Maria Heuss Dante Campregher2025≈ 78%
- Post-Hoc Concept Disentanglement: From Correlated to Isolated Concept RepresentationsSebastian Lapuschkin, Wojciech Samek, Frederik Pahde Eren Erogullari2025≈ 77%
- Short-circuiting Shortcuts: Mechanistic Investigation of Shortcuts in Text ClassificationShihan Wang, Antske Fokkens Leon Eshuijs2025≈ 77%
- Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in TransformersHenry Conklin, Yukang Yang, Thomas Griffiths, Jonathan Cohen, Sarah-Jane Leslie Andrew Nam2025≈ 77%
- Attention Gathers, MLPs Compose: A Causal Analysis of an Action-Outcome Circuit in VideoViTSai V R Chereddy2026≈ 77%
- Do Attention Heads Compete or Cooperate during Counting?\'Ad\'am Frakn\'oi, M\'at\'e Gedeon, Andr\'as Kornai, Zsolt Zombori P\'al Zs\'amboki2025≈ 77%
- Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AINooshin Bahador2025≈ 77%
- Measuring and modeling the perception of natural and unconstrained gaze in humans and machinesTao Gao, Nancy Kanwisher, Joshua Tenenbaum, Shimon Ullman Daniel Harari2016≈ 77%
- Disentangling Polysemantic Neurons with a Null-Calibrated Polysemanticity Index and Causal Patch InterventionsDhruv Kumar Manan Gupta2025≈ 76%
- Towards Understanding the Nature of Attention with Low-Rank Sparse DecompositionZhengfu He and Junxuan Wang and Rui Lin and Xuyang Ge and Wentao Shu and Qiong Tang and Junping Zhang and Xipeng Qiu2025≈ 76%
- ≈ 75%
- ≈ 75%
- ≈ 74%
- ≈ 74%
- ≈ 74%
- Collective intelligence: A unifying concept for integrating biology across scales and substratesin corpus2024≈ 74%
- ≈ 74%
- ≈ 74%
- ≈ 73%
- The Platonic Representation Hypothesisin corpus2024≈ 73%
- ≈ 73%
- ≈ 73%
- Toy Models of Superpositionin corpus2022≈ 73%