concept
active
concept:gelu-activation-functionGeLU Activation Function
The nonlinear activation function used in MLP layers; prevents the linearization approach used for attention layers from extending to MLP layers
Neighborhood — ranked by edge-count
Claims (1)
claim
- Key limitation of the paper's approach; MLP layers make up 2/3 of standard transformer parameters
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Prior Anthropic approach to increasing neuron monosemanticity via activation function design; found to make some neurons more interpretable at cost of others
- Internal representations of the model on which probes operate; the method uses activations to rank datapoints.
- The conventional approach (e.g., SAEs, transcoders) of decomposing activations into interpretable features.
- Technique of reading out model beliefs from internal activations before the final answer token is generated
- Internals-based approach to which the logit monitor is complementary.
- Seven binary classification tasks recast as natural-language yes/no questions.
- Latent model activations when processing inputs framed from another agent's perspective
- Using softmax to translate membrane potentials into firing rates, implementing lateral inhibition.