finding

active

finding:tem-t-memory-neurons-show-spatially-tuned-firing-resembling-hippocampal-place-cells-in-each-environment

TEM-t memory neurons show spatially-tuned firing resembling hippocampal place cells in each environment

Empirical result demonstrating that the sparse softmax activation of memory neurons produces place-cell-like spatial tuning.

Source paper

extracted_from

Relating transformers to models and neural representations of the hippocampal formation

(2021) · James C. R. Whittington · Joseph W. Warren · Timothy E.J. Behrens

Neighborhood — ranked by edge-count

Claims (1)

claim

Transformer memory neurons resemble hippocampal place cells due to sparse softmax activation producing spatial tuning
supports
Interpretation of why memory neurons in the biologically-instantiated transformer architecture acquire place-cell-like properties.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

TEM-t instantiates hippocampal indexing theory by using memory neurons to bind cortical representations across brain regionsclaim0.853
Theoretical claim linking the TEM-t architecture to the Teyler-Rudy hippocampal indexing theory.
Novel place cell metric (largest connected component firing mass ratio) successfully distinguishes TEM-t memory neurons (place cells) from RNN neurons (grid cells)finding0.815
Methodological validation result confirming the place-cell metric separates cell types in TEM-t.
TEM-t with linear activations learns grid-cell-like position encoding representations in 2D spatial environmentsfinding0.804
Empirical result showing TEM-t recapitulates entorhinal grid cell representations with linear post-transition activation.
TEM-t learns band-cell-like position encoding representations resembling Krupic et al. band cellsfinding0.780
Empirical result showing TEM-t position encodings also recapitulate band cells, not just grid cells.
TEM-t learns grid cells in hexagonal 6-connected worldsfinding0.774
Empirical extension showing grid cell learning generalises to non-4-connected spatial environments.
would place-like representations emerge in memory neurons for activation functions other than softmax?question0.763
Open empirical question left for future work about robustness of place cell emergence.
Short-term cortical memory neurons may suffice for transformer-like computation in cortex without hippocampal involvement for non-long-term-memory taskshypothesis0.755
Speculative hypothesis about how cortical transformer instantiation avoids requiring hippocampus.
TEM memory retrieval is mathematically equivalent to transformer self-attention without softmaxclaim0.755
Central theoretical claim: a single step of TEM attractor dynamics equals a dot-product attention, making TEM a special case of transformer.