finding
active
finding:without-repetition-transcoder-on-abs-problem-learns-almost-exactly-relu-x-i-and-relu-x-i-features-with-small-residualWithout repetition, transcoder on abs problem learns almost exactly ReLU(x_i) and ReLU(-x_i) features with small residual
Baseline toy experiment.
Neighborhood — ranked by edge-count
Papers (1)
paper
Hypotheses (1)
hypothesis
- Two necessary conditions.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Transcoders easily discover the ReLU(x)+ReLU(-x) solution by default on the absolute value problemfinding0.860Baseline result before repetition.
- Demonstrates unfaithfulness in a simple setup.
- Transcoders could cheat Jacobian matching via large-weight, very negative-bias features exploiting the L1/L0 gaphypothesis0.769Gradient masking risk; mitigated by tanh L1 or penalizing Jacobian contribution L1.
- Synthetic theoretical example showing pernicious divergence via hidden pathway activation
- ReLU/JumpReLU restrict blocks to a cone and fail to recover superpositions a signed code resolves.claim0.756Justifies no coordinate-wise nonlinearity.
- Without superposition.
- Core insight: reconstruction objective combined with appropriate initialization and KL regularization produces human-interpretable explanations as emergent property.
- Key improvement in cross-task generalization enabled by explicit instruction framing.