method
active
method:transcodersTranscoders
Decomposition method for activations; VPD is compared against transcoders in sparsity-reconstruction tradeoff.
Neighborhood — ranked by edge-count
Papers (6)
paper
Frameworks (2)
framework
- Cross-Layer TranscodersextendsMLP replacement with sparse features underlying the original attribution graphs.
- Attentional featuresanalogous_toProposed attention analog of sparse monosemantic features for decomposing attention layers.
Methods (5)
method
- Single-layer transcoderrelated_toUsed to express MLP in monosemantic features, 4096 features with JumpReLU.
- Transcoderrelated_toUses activations from one layer to predict the next; special case of crosscoders.
- Sparse Dictionary LearningextendsGeneral method for finding overcomplete sparse decompositions; the paper uses sparse autoencoders as an approximation
- Toy computation y=abs(x) implemented by two ReLU neurons, used to study unfaithfulness.
- Sparsity regularizer approximating L0 used to avoid duplicate features.
Claims (1)
claim
- Quantitative advantage claimed for VPD over a prior activation-decomposition method.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Learn a shared feature space across two models; basis of the work.
- Decomposition approach mixing frozen base attention patterns.
- Mask virtual weights during training targeting interference.
- Second category of SAE/transcoder concerns, distinct from faithfulness.
- Replacement model architecture with 30M features replacing MLP neurons.
- Neural network architecture that learns compressed representations; SOHMs are functionally equivalent.
- Discussion.