paper
active
paper:headvis-interactive-90fa42

HeadVis: An Interactive Tool For Investigating Attention Heads

ByAuthors

TL;DR

A single attention head's behavior on the full data distribution rarely matches what a narrow task suggests, and HeadVis, an interactive tool for browsing heads in Claude Haiku 3.5 and Gemma 3 1B, makes that gap visible. HeadVis combines a metric scatter plot of heads, attention patterns bucketed by max attention, PCA of Q/K/O activations, and QK/OV attributions over a 10M-feature weakly causal crosscoder. Four case studies show how far the workflow reaches. A fuzzy poetry induction head matches on 'ee' sounds in QK and copies literally in OV. An early-layer line width head turns out to be polysemantic: it handles years 1000–1986, multi-token words and newlines, and PCA separates all three behaviors. The answer selection head reuses one selection template well beyond multiple choice, but its PCA is a single cloud and the authors only guess it is monosemantic. A same-set suppression head sends a country's attention to the one foreign city in 98% of 185 sentences, with mean query-key cosine −0.47 against its own cities versus +0.01 elsewhere. Even so, it fits real dataset examples poorly. Heads can be polysemantic without superposition, since their outputs are high rank. This gives four obstacles to an attention analogue of transcoders: unknown attention biology, head polysemanticity, attention superposition and high-rank attentional features. The paper argues that these crisp, findable examples now give decomposition methods concrete targets.

What to take away

  1. 1. HeadVis pairs a configurable per-head metric scatter plot, attention patterns bucketed by max attention, PCA of Q/K/O activations, and real-time QK/OV feature attributions through a 10M-feature weakly causal crosscoder.
  2. 2. A fuzzy poetry induction head matches 'ee'-sound line endings in QK (a '-ea sound at line end' query feature against a 'newline after an ee sound' key feature) while its OV circuit copies exactly like literal induction, so the fuzziness lives entirely in QK.
  3. 3. An early-layer line width head in Claude Haiku 3.5 is polysemantic: it attends between the tokens of years such as 1776 (split into 1 + 776), between pieces of multi-token words, and between newlines, and PCA of Q, K and O separates all three behaviors into clusters.
  4. 4. Sweeping the year in a fixed prompt shows the line width head attends strongly only for 1000–1986, a constraint that neither the attention patterns nor the per-example attributions revealed.
  5. 5. The answer selection head's PCA is one continuous cloud with no clusters, so the line width head's PCA check cannot diagnose its polysemanticity.
  6. 6. Projecting the 'correct answer' feature through the answer selection head's key and query weights lands on other 'this is the relevant content' features that are near-orthogonal in the residual stream, whereas the line width head's years feature lands only near other year features. This supports a single general selection behavior.
  7. 7. In 185 sentences of the form '{five cities} are all cities in {country}', the same-set suppression head attends more to the one foreign city than to any of the four native cities 98% of the time.
  8. 8. Mean-subtracted country query vectors have mean cosine −0.47 with their own cities' key vectors and +0.01 with other cities, even though the residual-stream cosine is +0.17, so W_Q and W_K flip the sign.
  9. 9. Replacing 'Ivory Coast' with 'Zambia' in a sequence about a Zambian nonprofit dropped attention to the token 'a' from 0.87 to 0.08, and the median across the other 51 African countries was 0.27. An earlier head's K-composition writes the Zambian features onto 'a'.
  10. 10. Open question: can an attention layer be decomposed into sparse 'attentional features' that are possibly high rank, with the line width head giving three separate features and the answer selection head one high-rank feature, and is attention superposition real in any LLM? No confirmed example exists yet.

Peer brief — for seminar discussion

HeadVis is an open-source interactive tool, the attention-head analogue of feature visualization, for studying heads over a full data distribution rather than a single task. It combines a scatter plot of per-head metrics such as an induction score, top-activating attention patterns, PCA of Q/K/O activations, and real-time QK/OV attributions through a 10M-feature weakly causal crosscoder. The paper applies it to heads in Claude Haiku 3.5 and releases a demo on Gemma 3 1B. Four case studies run from easy to hard. A fuzzy induction head matches in feature space (an 'ee' rhyme) in QK and copies literally in OV. A line width head turns out to be cleanly polysemantic, with year, multi-token-word and newline behaviors that separate in PCA, and its year behavior only fires for 1000–1986. The answer selection head appears to reuse one 'relevant content' selection template across multiple choice, sentiment and dialogue, although PCA shows no clusters. A same-set suppression head sends a country's attention to the one foreign city in 98% of 185 synthetic sentences, with a −0.47 mean QK cosine against its own cities. The load-bearing finding is the pattern across the four: narrow-task behavior underdetermines what a head does, and the hard cases need feature-level tools because token patterns mislead. In the Ivory Coast example, attention depends on a Zambian feature that an earlier head wrote onto an unrelated token. The implication is that attention needs its own decomposition, and the discussion names four obstacles: attention biology, head polysemanticity without superposition, attention superposition, and high-rank features. These are compared with existing rank-1 OV approaches such as attention-output SAEs and Lorsa. The paper predicts that a good decomposition would split the line width head into three features and keep the answer selection head as one. A critical reader would push back on evidence strength and on how representative the heads are. The monosemanticity claim for answer selection rests on a handful of examples and virtual-weight neighbors, with no text hypothesis tested against a grader. The suppression mechanism was shown mainly on synthetic prompts and fits real data loosely. The heads were also chosen because they were interpretable, so the generalization from them is unclear. PCA, which the authors call hypothesis generation, can look clean or fail for reasons unrelated to how many behaviors a head has. A more direct alternative would be to split the head into separate heads and check that they reproduce its output. Another would be ablation or causal patching, but the paper uses neither.

Methods (13)

Frameworks (5)

Datasets (2)

Hypotheses (3)

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar