finding
active
finding:most-crosscoder-features-peak-in-a-particular-layer-and-decay-gradually-many-with-substantial-norm-across-most-or-all-layersMost crosscoder features peak in a particular layer and decay gradually, many with substantial norm across most or all layers
Decoder norm profile across 18 layers.
Source paper
extracted_fromNeighborhood — ranked by edge-count
Hypotheses (1)
hypothesis
- Consistent but alternative explanations like amplification exist.
Findings (1)
finding
- Eval loss interpolates fairly linearly as convolutional window K varies from 1 to n_layers, with no inflection pointassociated_withLocally masked crosscoder equals smaller FLOP-matched global crosscoder.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Cross-layer transcoder features form a mix of local, global and intermediate features by decoder normfinding0.850Preliminary decoder norm analysis.
- Motivation for DFC.
- Footnote 2.
- Appendix C.4.
- Preliminary observation.
- Decoder cost dominates.
- Motivating example with 13 vs 3 features.
- Discussion.