method
active
method:transcoders

Transcoders

Decomposition method for activations; VPD is compared against transcoders in sparsity-reconstruction tradeoff.

Neighborhood — ranked by edge-count

Frameworks (2)

framework
  • MLP replacement with sparse features underlying the original attribution graphs.
  • Proposed attention analog of sparse monosemantic features for decomposing attention layers.

Methods (5)

method
  • Used to express MLP in monosemantic features, 4096 features with JumpReLU.
  • Transcoder
    related_to
    Uses activations from one layer to predict the next; special case of crosscoders.
  • General method for finding overcomplete sparse decompositions; the paper uses sparse autoencoders as an approximation
  • Toy computation y=abs(x) implemented by two ReLU neurons, used to study unfaithfulness.
  • Sparsity regularizer approximating L0 used to avoid duplicate features.

Claims (1)

claim

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.