Position Encodings

Mechanism for encoding sequence order in transformers; paper argues these should reflect learned structural representations rather than fixed sines/cosines.

Neighborhood — ranked by edge-count

Claims (1)

claim

Position encodings should represent location in a learned structure inferred on the fly rather than fixed sines and cosines
extends
Novel interpretive claim about position encodings inspired by the TEM-transformer correspondence.

Methods (1)

method

Recurrent Position Encodings
extends
Key modification to transformers proposed in this paper: position encodings generated by a recurrent network trained on action sequences.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Grammar as Positional Encoding for Languagehypothesis0.848
Hypothesis that in language tasks, the abstract structure encoded in positional encodings corresponds to grammatical structure.
Positional encodings inferred on the fly from previously learned structures would offer fruitful research direction for language, maths, and logicclaim0.819
Forward-looking interpretive claim about the implications of recurrent position encodings for NLP research.
Positional Informationconcept0.807
The biological mechanism by which chemical gradients create developmental patterns; cited as insufficient to explain coherence of embryological unfolding as a whole
Positional information gradientconcept0.797
Gradient that tells a cell its correct position; stress arises from deviation from this gradient.
what is the analogue of spatial positional encodings for higher order tasks such as language?question0.796
Open question raised in Discussion about extending TEM-t principles beyond spatial navigation.
Precision Encodingconcept0.775
Quantifies certainty in beliefs; modulates policy selection; related to dopamine encoding.
position of a roomconcept0.765
The location of a room within the building in relation to movement, light, and connection to the outdoors; the first stage of unfolding.
Sensory Landmark Position Encoding Stabilizationmethod0.765
Method for stabilising drifting recurrent position encodings by querying stored landmark memories to correct path-integrated position.