finding
active
finding:relu-makes-negative-interference-free-in-the-1-sparse-caseReLU makes negative interference free in the 1-sparse case
Explains preference for negative interference.
Neighborhood — ranked by edge-count
Papers (1)
paper
- Toy Models of Superpositionmentions
Claims (1)
claim
- Explanation for neuron interpretability.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Exception to ReLU underperformance.
- ReLU hidden layer model circumvents the ReLU by using positive biases or simulating biasfinding0.752Limitation of that toy model.
- ReLU/JumpReLU restrict blocks to a cone and fail to recover superpositions a signed code resolves.claim0.739Justifies no coordinate-wise nonlinearity.
- Baseline toy experiment.
- Better bases may give sparsity.
- ReLU hidden layer model (n=10, m=5) shifts from monosemantic to polysemantic neurons as sparsity increasesfinding0.716Neuron-level phase change.
- Synthetic theoretical example showing pernicious divergence via hidden pathway activation
- Author's conclusion after extensive investigation of architectural approaches to monosemanticity