claim
active
claim:relu-jumprelu-restrict-blocks-to-a-cone-and-fail-to-recover-superpositions-a-signed-code-resolvesReLU/JumpReLU restrict blocks to a cone and fail to recover superpositions a signed code resolves.
Justifies no coordinate-wise nonlinearity.
Source paper
extracted_from(2026) · Thomas Fel · Matthew Kowal · Mozes Jacobs · Dron Hazra +21
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Rotation-invariance argument.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Design justification.
- Baseline toy experiment.
- Core demonstration.
- Transcoders easily discover the ReLU(x)+ReLU(-x) solution by default on the absolute value problemfinding0.744Baseline result before repetition.
- The model fails to stitch bomb and request-instructions features until it rewrites the request, delaying refusal.hypothesis0.742Validated by activating a make-a-bomb feature on BOMB token.
- Explains preference for negative interference.
- Synthetic theoretical example showing pernicious divergence via hidden pathway activation
- ReLU hidden layer model circumvents the ReLU by using positive biases or simulating biasfinding0.730Limitation of that toy model.