Position encodings should represent location in a learned structure inferred on the fly rather than fixed sines and cosines

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Positional encodings inferred on the fly from previously learned structures would offer fruitful research direction for language, maths, and logicclaim0.881
  Forward-looking interpretive claim about the implications of recurrent position encodings for NLP research.
• Recurrent Position Encodingsmethod0.796
  Key modification to transformers proposed in this paper: position encodings generated by a recurrent network trained on action sequences.
• TEM-t learns band-cell-like position encoding representations resembling Krupic et al. band cellsfinding0.787
  Empirical result showing TEM-t position encodings also recapitulate band cells, not just grid cells.
• Grammar as Positional Encoding for Languagehypothesis0.777
  Hypothesis that in language tasks, the abstract structure encoded in positional encodings corresponds to grammatical structure.
• what is the analogue of spatial positional encodings for higher order tasks such as language?question0.771
  Open question raised in Discussion about extending TEM-t principles beyond spatial navigation.
• Neural Representations of Location Composed of Spatially Periodic Bands (Krupic et al., 2012)concept0.764
  Discovery of band cells; TEM-t also recapitulates these representations.
• Learning to encode position for transformer with continuous dynamical model (Liu et al., 2020)concept0.760
  Prior work on learned dynamic position encodings; cited alongside Wang et al. as precedent.
• Assuming linear representations enables identifying the location of certain variables in a DNN, but many insights fail to generalise when more powerful non-linear maps are usedclaim0.747
  Interpretive claim about what linear DAS results actually tell us