Recurrent Position Encodings

Key modification to transformers proposed in this paper: position encodings generated by a recurrent network trained on action sequences.

Neighborhood — ranked by edge-count

paper

framework

TEM-Transformer (TEM-t)
implements
The transformer version directly analogous to TEM, introduced in this paper, offering dramatic performance improvements.

concept

Path Integration
analogous_to
Neural mechanism for tracking location through accumulation of self-movement vectors; shown to play the role of position encodings in TEM.
Position Encodings
extends
Mechanism for encoding sequence order in transformers; paper argues these should reflect learned structural representations rather than fixed sines/cosines.

method

Sensory Landmark Position Encoding Stabilization
uses
Method for stabilising drifting recurrent position encodings by querying stored landmark memories to correct path-integrated position.

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Position encodings should represent location in a learned structure inferred on the fly rather than fixed sines and cosinesclaim0.796
Novel interpretive claim about position encodings inspired by the TEM-transformer correspondence.
Positional encodings inferred on the fly from previously learned structures would offer fruitful research direction for language, maths, and logicclaim0.789
Forward-looking interpretive claim about the implications of recurrent position encodings for NLP research.
Grammar as Positional Encoding for Languagehypothesis0.787
Hypothesis that in language tasks, the abstract structure encoded in positional encodings corresponds to grammatical structure.
what is the analogue of spatial positional encodings for higher order tasks such as language?question0.764
Open question raised in Discussion about extending TEM-t principles beyond spatial navigation.
repetitionconcept0.754
The reappearance of similar elements, essential for unity and order in a living floor or ceiling.
Recurrent Logic Circuitconcept0.747
The key novel property of DiffLogic CA — logic gate networks that are recurrent both spatially and temporally
Precision Encodingconcept0.737
Quantifies certainty in beliefs; modulates policy selection; related to dopamine encoding.
Positional Informationconcept0.735
The biological mechanism by which chemical gradients create developmental patterns; cited as insufficient to explain coherence of embryological unfolding as a whole