finding

active

finding:identification-of-algorithms-implemented-in-attention-layers-distributed-across-attention-heads

Identification of algorithms implemented in attention layers, distributed across attention heads

VPD successfully recovered interpretable attention algorithms (previous-token behavior, syntax-boundary routing) in weight space without requiring manual decomposition across heads.

Source paper

extracted_from

cimcWhitepaper

Neighborhood — ranked by edge-count

Hypotheses (1)

hypothesis

Language models contain interpretable computational structure encoded in their parameter weights, not irreducibly impenetrable complexity
answered_by
Core empirical hypothesis of the paper, supported by successful VPD decomposition yielding ~10,000 interpretable subcomponents across 24 weight matrices.

Communities (4)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
Mechanistic structure of transformer attention computations
members_of
Identifies distributed algorithms implemented across attention heads, with focus on causal masking limitations and emergent capabilities via activation manifold steering.
Distributed attention head decomposition
members_of
Mechanistic interpretability approach decomposing attention heads into query/key subcomponents with distinct algorithmic roles
Emergence through distributed attention and uncertainty
members_of
Explores how complex phenomena arise from non-linear interactions across distributed systems, emphasizing productive not-knowing and implicit learning mechanisms.

Methods (1)

method

Adversarial Parameter Decomposition (VPD)
supports
Core technique introduced in this paper for decomposing neural network weight matrices into mechanistically simple, interpretable rank-one subcomponents.

Questions (1)

question

How can mechanistic interpretability methods automatically identify attention computations that span multiple attention heads?
answered_by
Long-standing bottleneck in mechanistic interpretability that VPD addresses by working natively on attention weight matrices.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Attention algorithms are usually distributed across attention headsclaim0.890
Claim supported by VPD's recovery of cross-head attention subcomponents, noted in footnote.
Attention computations distribute across heads via parameter subcomponents with interpretable rolesfinding0.809
Mechanistic discovery about how attention mechanisms decompose into interpretable parameter components.
attention computationconcept0.797
Process using Q, K, V to compute a heat map over K and weighted sum of V.
Specific architectural components (attention heads, FFN layers) are responsible for encoding deception and task semanticshypothesis0.790
Future work direction: mechanistic interpretability to identify precise components encoding deception
One-layer model attention heads encode Python-specific skip-trigrams including indentation-based elif/else prediction and function signature patternsfinding0.787
Concrete example from examining expanded QK/OV matrices showing how specific programming language structure is encoded in attention weights
A pair of query and key subcomponents distributed across attention heads performs syntax-boundary routingfinding0.784
VPD recovers an attention algorithm for routing across syntactic boundaries, distributed across heads.
Each attention head has two largely independent computations: a QK circuit computing the attention pattern and an OV circuit computing the effect if attended toclaim0.784
Key decomposition enabling separate analysis of where attention goes and what it does
Most attention heads in one-layer models dedicate an enormous fraction of their capacity to copying behaviorclaim0.782
Empirical observation from examining expanded OV/QK matrices; approximately 10 out of 12 heads show significant copying