claim
active
claim:attention-scores-are-a-bilinear-function-of-residual-stream-at-query-and-key-positions-so-can-be-written-as-sums-of-feature-feature-dot-productsAttention scores are a bilinear function of residual stream at query and key positions, so can be written as sums of feature-feature dot products
Core insight.
Source paper
extracted_fromNeighborhood — ranked by edge-count
Papers (1)
paper
Methods (1)
method
- QK AttributionssupportsCore method decomposing attention scores as bilinear sums of query-key feature pairs.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Mathematical equivalence enabling independent analysis of each attention head
- Mathematical equivalence showing the relationship between attention mechanisms and convolutional operations
- Justifies the methodological choice of attention over concatenation, mean pooling, residual connections, or joint embedding.
- Limitation of single-prompt selection.
- Reason for adopting the third strategy.
- Fig. 13.
- Formal bound showing attention patterns change slowly when residual stream converges, linking fixed points to stage stability
- Per-head metric measuring fraction of attention weight concentrated on first token position