Softmax Activation Function as Neuronal Model

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• The softmax function may be interpreted as performing lateral inhibition, leading to sharper inferences.claim0.801
  Interpretation of neural implementation in Section 5.1.
• Softmax Functionmethod0.787
  Neuronal dynamics computed from free energy gradients; interpreted as average firing rate of neural populations.
• would place-like representations emerge in memory neurons for activation functions other than softmax?question0.786
  Open empirical question left for future work about robustness of place cell emergence.
• Winner take-all architectures of decision-making are already commonplace in computational neuroscience, and the softmax function provides a smooth approximation.claim0.752
  Neural plausibility argument for softmax policy selection.
• Softmax policy selectionmethod0.726
  Selecting policies using a softmax (normalized exponential) function of negative expected free energy.
• Models with 1-hot activation sparsity still have polysemantic neurons; single neuron trained on 4 mutually exclusive features prefers polysemantic representation with loss ~0.7 vs 0.8finding0.725
  Counter-example disproving that architectural sparsity alone can prevent polysemanticity
• All neuronal processing and action selection minimize variational free energy, unifying perception, action, and learning.claim0.725
  Fundamental assertion: single imperative (free energy minimization) explains diverse cognitive and neural phenomena.
• Training models with sparse activations cannot fully prevent polysemanticity because cross-entropy loss creates incentives for polysemantic neurons even without superpositionclaim0.715
  Author's conclusion after extensive investigation of architectural approaches to monosemanticity