claim
active
claim:cross-layer-transcoders-are-likely-not-the-best-long-term-abstraction-the-method-is-a-stepping-stoneCross-layer transcoders are likely not the best long-term abstraction; the method is a stepping stone.
Discussion.
Neighborhood — ranked by edge-count
Papers (1)
paper
Methods (1)
method
- Replacement model architecture with 30M features replacing MLP neurons.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- MLP replacement with sparse features underlying the original attribution graphs.
- Cross-layer transcoder features form a mix of local, global and intermediate features by decoder normfinding0.815Preliminary decoder norm analysis.
- Mask virtual weights during training targeting interference.
- Used to express MLP in monosemantic features, 4096 features with JumpReLU.
- Motivation for DFC.
- Features smeared across layers cannot be fully disentangled by SAE on a single residual stream.
- Decoder norm profile across 18 layers.
- Compared with base SAE transfer.