finding
active
finding:crosscoders-are-about-2x-less-flop-efficient-than-per-layer-saes-at-equal-eval-loss-at-large-computeCrosscoders are about 2x less FLOP-efficient than per-layer SAEs at equal eval loss at large compute
Cost of identifying cross-layer structure.
Source paper
extracted_fromRelated by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Indicates redundant structure across layers.
- Naive implementation results.
- Decoder cost dominates.
- Crosscoders give a dramatically better whole-model (MSE, L0) tradeoff than summed per-layer SAEsfinding0.835Consolidating shared structure gives concise decomposition.
- Motivating example with 13 vs 3 features.
- Motivation for DFC.
- Circuit simplification theory.
- Decoder norm profile across 18 layers.