method
active
method:frobenius-norm-composition-measurementFrobenius Norm Composition Measurement
Measuring Q-, K-, V-composition between attention heads by computing the Frobenius norm of the product of relevant matrices divided by norms of individual matrices
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- K-CompositionaboutA form of attention head composition where W_K reads from a subspace affected by a previous head; central to how induction heads are implemented
Methods (1)
method
- Frobenius Norm Comparisonrelated_toUsed to compare attention matrix similarity across recurrences and validate cyclic fixed point behavior
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Formal linguistic principle: meaning of a whole depends only on meanings of parts and how they are fitted together (bottom-up meaning flow).
- Sparsity penalty weighting by sum of per-layer decoder norms, enabling comparability with per-layer SAEs.
- Result from applying the Frobenius norm composition measurement to all attention head pairs in the two-layer model
- Post-hoc metric for model exclusivity in standard crosscoders.
- Role of extra dimensions.
- Scaling aggregated gradient by the maximum gradient norm among tasks.
- Alternative penalty that spreads features across layers and finds only shared features.
- Applied to 30-dim (layer, model) norm vectors to find shared structure across scale.