concept
active
concept:weight-superposition

Weight superposition

Virtual weights between features forced into superposition by low-dimensional projections.

Neighborhood — ranked by edge-count

Concepts (2)

concept
  • Superposition
    extendsrelated_to
    Phenomenon where models represent more features than dimensions via almost-orthogonal directions.
  • Interference Weights
    associated_with
    Logit weight contributions from a feature that arise due to superposition with other features, not from the feature's own causal role

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

  • More computations than neurons/dimensions.
  • Attentional features spread across multiple heads.
  • Any method enabling enumeration of features.
  • Superposition configuration associated with a plateau and a dip in vulnerability.
  • Core theoretical framework: neural networks represent more features than neurons by encoding features as directions in superposition
  • Specific phrases or sequences memorized via binary features in superposition, enabling narrow pattern matching despite few neurons
  • Equal Weightingframework0.778
    Baseline MTL approach minimizing sum of task losses with equal weights; suffers from task balancing
  • Weight spaceconcept0.777
    The space of the model's parameter matrices, where VPD operations take place.