Anti-correlated Heads

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• We hypothesize that anti-correlated heads may play a functional role in maintaining coherency by counteracting excessive stylistic shiftshypothesis0.773
  Motivates including Head Cor+Anti condition in steering position comparison
• Concordance headsconcept0.739
  QK circuit heads hypothesized to measure likelihood of an output given prior activations, used in prefill detection.
• Attention headsconcept0.721
  Transformer attention heads that could be recruited to extract different kinds of information (text vs. thoughts).
• Induction Headsconcept0.714
  Mechanistic circuits in transformers documented by Olsson et al. 2022, cited as evidence for pattern-repository assumption
• attention head localization analysismethod0.696
  Analysis measuring whether each attention head's maximum attention increase points to the correct injected sentence
• Attention computations distribute across heads via parameter subcomponents with interpretable rolesfinding0.686
  Mechanistic discovery about how attention mechanisms decompose into interpretable parameter components.
• Attention heads can be understood as independent operations each adding their output to the residual stream, equivalent to the concatenate-and-multiply formulationclaim0.680
  Mathematical equivalence enabling independent analysis of each attention head
• Antipodal alignment between related datasets (e.g., larger_than and smaller_than) in smaller models resolves to common-direction alignment in larger modelsclaim0.679
  Scale-dependent structural finding from PCA visualizations in §4