framework
active
framework:relu-hidden-layer-toy-modelReLU hidden layer toy model
Toy model with privileged basis.
Neighborhood — ranked by edge-count
Papers (1)
paper
- Toy Models of Superpositionintroducesmentions
Methods (1)
method
- Visualizes polysemantic neurons.
Frameworks (1)
framework
- Model computing abs(x) in superposition.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- x'=ReLU(W^TWx+b), main model demonstrating superposition.
- ReLU hidden layer model circumvents the ReLU by using positive biases or simulating biasfinding0.804Limitation of that toy model.
- ReLU hidden layer model (n=10, m=5) shifts from monosemantic to polysemantic neurons as sparsity increasesfinding0.776Neuron-level phase change.
- Core demonstration.
- Open question.
- Synthetic two-model activations built from shared and exclusive latent factors with controllable ground truth.
- Situates the model in Rochat's developmental framework