concept
active
concept:attention-superpositionAttention superposition
Attentional features spread across multiple heads.
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (3)
concept
- Superpositionrelated_toPhenomenon where models represent more features than dimensions via almost-orthogonal directions.
- Attention head superpositionrelated_toPossibility that functional units are distributed across heads.
- Attention biologyassociated_withStudy of what attentional features and head behaviors exist.
Hypotheses (1)
hypothesis
- Suspected but no confirmed example in real LLM.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Specific phrases or sequences memorized via binary features in superposition, enabling narrow pattern matching despite few neurons
- Process using Q, K, V to compute a heat map over K and weighted sum of V.
- Virtual weights between features forced into superposition by low-dimensional projections.
- More computations than neurons/dimensions.
- Theoretical model of how neural networks encode more features than dimensions, informing linear representation work.
- Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
- Any method enabling enumeration of features.
- Blocking attention edges to trace information flow.