claim
active
claim:crosscoders-bake-in-clustering-of-features-across-layers-at-training-time-that-is-hard-to-do-post-hoc-on-saesCrosscoders bake in clustering of features across layers at training time that is hard to do post hoc on SAEs
Due to SAE training stochasticity.
Source paper
extracted_fromNeighborhood — ranked by edge-count
Findings (1)
finding
- Crosscoders give a dramatically better whole-model (MSE, L0) tradeoff than summed per-layer SAEssupportsConsolidating shared structure gives concise decomposition.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Conclusion.
- Circuit simplification theory.
- Motivating example with 13 vs 3 features.
- Indicates redundant structure across layers.
- Crosscoders are about 2x less FLOP-efficient than per-layer SAEs at equal eval loss at large computefinding0.777Cost of identifying cross-layer structure.
- Decoder cost dominates.
- Crosscoder model diffing is best used as a high-recall pre-screening tool followed by verification.claim0.776Recommended workflow.
- Motivation for DFC.