claim

active

claim:tem-s-path-integration-representation-g-plays-the-role-of-position-encodings-in-transformers

TEM's path-integration representation g plays the role of position encodings in transformers

Key structural correspondence claim linking the neuroscience model's spatial representation to ML concept of position encoding.

Source paper

extracted_from

Relating transformers to models and neural representations of the hippocampal formation

(2021) · James C. R. Whittington · Joseph W. Warren · Timothy E.J. Behrens

Neighborhood — ranked by edge-count

Papers (1)

paper

Relating transformers to models and neural representations of the hippocampal formation
introduces

Findings (1)

finding

TEM-t with linear activations learns grid-cell-like position encoding representations in 2D spatial environments
supports
Empirical result showing TEM-t recapitulates entorhinal grid cell representations with linear post-transition activation.

Claims (1)

claim

TEM memory retrieval is mathematically equivalent to transformer self-attention without softmax
extends
Central theoretical claim: a single step of TEM attractor dynamics equals a dot-product attention, making TEM a special case of transformer.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

The direct path W_U W_E in larger transformers represents bigram statistics not captured by more general grammatical rulesclaim0.764
Interpretation of the role of the direct path in multi-layer transformers; e.g. encoding that 'Barack' is often followed by 'Obama'
TEM-t learns band-cell-like position encoding representations resembling Krupic et al. band cellsfinding0.759
Empirical result showing TEM-t position encodings also recapitulate band cells, not just grid cells.
Learning to encode position for transformer with continuous dynamical model (Liu et al., 2020)concept0.756
Prior work on learned dynamic position encodings; cited alongside Wang et al. as precedent.
Different introspective tasks may preferentially use different path distributions in the transformer.claim0.750
Interpretive claim connecting exponential path combinatorics to Lindsey's layer-dependent findings.
Position encodings should represent location in a learned structure inferred on the fly rather than fixed sines and cosinesclaim0.742
Novel interpretive claim about position encodings inspired by the TEM-transformer correspondence.
TEM-t instantiates hippocampal indexing theory by using memory neurons to bind cortical representations across brain regionsclaim0.733
Theoretical claim linking the TEM-t architecture to the Teyler-Rudy hippocampal indexing theory.
Redundant information paths create interference patterns, so transformers likely experience memory and cognition as interferometric and continuous.claim0.733
Janus's claim linking path redundancy to interferometric phenomenology.
The last layer of the transformer has the largest projection magnitude on the reflection direction, likely because it directly controls generation of reflection keywordsclaim0.730
Interpretive claim from attention head attribution analysis in appendix