concept
active
concept:gelu-activation-functionGeLU Activation Function
The nonlinear activation function used in MLP layers; prevents the linearization approach used for attention layers from extending to MLP layers
Neighborhood — ranked by edge-count
Claims (1)
claim
- Key limitation of the paper's approach; MLP layers make up 2/3 of standard transformer parameters
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Prior Anthropic approach to increasing neuron monosemanticity via activation function design; found to make some neurons more interpretable at cost of others
- Internal representations of the model on which probes operate; the method uses activations to rank datapoints.
- The conventional approach (e.g., SAEs, transcoders) of decomposing activations into interpretable features.
- Technique of reading out model beliefs from internal activations before the final answer token is generated
- Latent model activations when processing inputs framed from another agent's perspective
- Using softmax to translate membrane potentials into firing rates, implementing lateral inhibition.
- Experimental technique to induce bioelectric state changes and measure consequences for collective decision-making (morphogenesis, cancer, organ formation).
- A failure mode where weak-tier models fail to invoke relevant harness artifacts (e.g., skills) during task solving