finding

active

finding:tem-t-with-linear-activations-learns-grid-cell-like-position-encoding-representations-in-2d-spatial-environments

TEM-t with linear activations learns grid-cell-like position encoding representations in 2D spatial environments

Empirical result showing TEM-t recapitulates entorhinal grid cell representations with linear post-transition activation.

Source paper

extracted_from

Relating transformers to models and neural representations of the hippocampal formation

(2021) · James C. R. Whittington · Joseph W. Warren · Timothy E.J. Behrens

Neighborhood — ranked by edge-count

Claims (2)

claim

TEM memory retrieval is mathematically equivalent to transformer self-attention without softmax
supports
Central theoretical claim: a single step of TEM attractor dynamics equals a dot-product attention, making TEM a special case of transformer.
TEM's path-integration representation g plays the role of position encodings in transformers
supports
Key structural correspondence claim linking the neuroscience model's spatial representation to ML concept of position encoding.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

TEM-t learns band-cell-like position encoding representations resembling Krupic et al. band cellsfinding0.856
Empirical result showing TEM-t position encodings also recapitulate band cells, not just grid cells.
TEM-t learns grid cells in hexagonal 6-connected worldsfinding0.845
Empirical extension showing grid cell learning generalises to non-4-connected spatial environments.
TEM-t instantiates hippocampal indexing theory by using memory neurons to bind cortical representations across brain regionsclaim0.804
Theoretical claim linking the TEM-t architecture to the Teyler-Rudy hippocampal indexing theory.
TEM-t memory neurons show spatially-tuned firing resembling hippocampal place cells in each environmentfinding0.804
Empirical result demonstrating that the sparse softmax activation of memory neurons produces place-cell-like spatial tuning.
Emergence of grid-like representations by training recurrent neural networks to perform spatial localization (Cueva & Wei, 2018)concept0.780
RNN model recapitulating grid cells; related work category 4.
Novel place cell metric (largest connected component firing mass ratio) successfully distinguishes TEM-t memory neurons (place cells) from RNN neurons (grid cells)finding0.758
Methodological validation result confirming the place-cell metric separates cell types in TEM-t.
Positional encodings inferred on the fly from previously learned structures would offer fruitful research direction for language, maths, and logicclaim0.747
Forward-looking interpretive claim about the implications of recurrent position encodings for NLP research.
TEM-t requires less time per gradient step than TEMfinding0.745
Empirical computational efficiency result comparing TEM-t to the original TEM implementation.