method
active
method:crosscoder-l1-penalty-with-per-model-summed-decoder-normsCrosscoder L1 penalty with per-model summed decoder norms
Key design choice encouraging feature exclusivity.
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Features writing predominantly to one model.
Methods (1)
method
- Train a single SAE encoding and decoding activations from two models simultaneously to find shared and exclusive features.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Footnote 2.
- Crosscoders give a dramatically better whole-model (MSE, L0) tradeoff than summed per-layer SAEsfinding0.773Consolidating shared structure gives concise decomposition.
- L1-of-norms is preferable for loss comparability and exposing layer- or model-specific featuresclaim0.762All experiments used L1-of-norms.
- Appendix C.4.
- Sparsity penalty weighting by sum of per-layer decoder norms, enabling comparability with per-layer SAEs.
- The objective function combining L2 reconstruction error and L1 penalty scaled by decoder norm, used to train the SAE.
- Adaptation of logit attribution to CLT features that decode into all subsequent layers.
- Decoder norm profile across 18 layers.