finding
active
finding:attention-head-outputs-affect-the-character-count-representation-4x-more-than-mlpsAttention head outputs affect the character count representation 4x more than MLPs
Justifies focusing on attention.
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Distributed algorithm.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Architectural rationale for why Style Modulation Heads are in attention layers rather than MLPs
- Quantitative result from eigenvalue analysis of expanded OV matrices; confirmed by qualitative inspection
- Confirms attention rather than MLP as the locus of persona generation
- Theoretical point.
- Empirical observation from examining expanded OV/QK matrices; approximately 10 out of 12 heads show significant copying
- Hypothesis based on observed negative cosine similarity between input and output weights of some neurons
- Suspected but no confirmed example in real LLM.
- Prediction for future decomposition.