concept
active
concept:attention-superposition

Attention superposition

Attentional features spread across multiple heads.

Neighborhood — ranked by edge-count

Concepts (3)

concept
  • Superposition
    related_to
    Phenomenon where models represent more features than dimensions via almost-orthogonal directions.
  • Possibility that functional units are distributed across heads.
  • Attention biology
    associated_with
    Study of what attentional features and head behaviors exist.

Hypotheses (1)

hypothesis

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

  • Specific phrases or sequences memorized via binary features in superposition, enabling narrow pattern matching despite few neurons
  • Process using Q, K, V to compute a heat map over K and weighted sum of V.
  • Virtual weights between features forced into superposition by low-dimensional projections.
  • More computations than neurons/dimensions.
  • Theoretical model of how neural networks encode more features than dimensions, informing linear representation work.
  • Attentionconcept0.798
    Selective attention mechanism is a component of GWT-2 indicator; AST-1 models and enables control over attention
  • Any method enabling enumeration of features.
  • Blocking attention edges to trace information flow.