finding
active
finding:eval-loss-interpolates-fairly-linearly-as-convolutional-window-k-varies-from-1-to-n-layers-with-no-inflection-pointEval loss interpolates fairly linearly as convolutional window K varies from 1 to n_layers, with no inflection point
Locally masked crosscoder equals smaller FLOP-matched global crosscoder.
Source paper
extracted_fromNeighborhood — ranked by edge-count
Findings (1)
finding
- Most crosscoder features peak in a particular layer and decay gradually, many with substantial norm across most or all layersassociated_withDecoder norm profile across 18 layers.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Indicates redundant structure across layers.
- L1-of-norms is preferable for loss comparability and exposing layer- or model-specific featuresclaim0.746All experiments used L1-of-norms.
- Basis of phenomenological argument.
- Crosscoders are about 2x less FLOP-efficient than per-layer SAEs at equal eval loss at large computefinding0.742Cost of identifying cross-layer structure.
- Smoothed behavior.
- Feature attribution (gradient-based) correlates 0.8 with ablation effects on the 'John' and 'Kobe' examples.finding0.734Validation of attribution as a fast proxy for causal importance.
- Limitation.