finding
active
finding:crosscoders-substantially-outperform-per-layer-saes-on-eval-loss-at-fixed-total-feature-count-on-18-layer-modelCrosscoders substantially outperform per-layer SAEs on eval loss at fixed total feature count on 18-layer model
Indicates redundant structure across layers.
Source paper
extracted_fromNeighborhood — ranked by edge-count
Claims (1)
claim
- Interpretation of eval loss result.
Datasets (1)
dataset
- Activations used to compare crosscoder vs per-layer SAEs.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Crosscoders are about 2x less FLOP-efficient than per-layer SAEs at equal eval loss at large computefinding0.877Cost of identifying cross-layer structure.
- Crosscoders give a dramatically better whole-model (MSE, L0) tradeoff than summed per-layer SAEsfinding0.852Consolidating shared structure gives concise decomposition.
- Naive implementation results.
- Motivating example with 13 vs 3 features.
- Decoder cost dominates.
- Locally masked crosscoder equals smaller FLOP-matched global crosscoder.
- Decoder norm profile across 18 layers.
- Circuit simplification theory.