claim

active

claim:positional-encodings-inferred-on-the-fly-from-previously-learned-structures-would-offer-fruitful-research-direction-for-language-maths-and-logic

Positional encodings inferred on the fly from previously learned structures would offer fruitful research direction for language, maths, and logic

Forward-looking interpretive claim about the implications of recurrent position encodings for NLP research.

Source paper

extracted_from

Relating transformers to models and neural representations of the hippocampal formation

(2021) · James C. R. Whittington · Joseph W. Warren · Timothy E.J. Behrens

Neighborhood — ranked by edge-count

Papers (1)

paper

Relating transformers to models and neural representations of the hippocampal formation
introduces

Hypotheses (1)

hypothesis

Grammar as Positional Encoding for Language
associated_with
Hypothesis that in language tasks, the abstract structure encoded in positional encodings corresponds to grammatical structure.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Position encodings should represent location in a learned structure inferred on the fly rather than fixed sines and cosinesclaim0.881
Novel interpretive claim about position encodings inspired by the TEM-transformer correspondence.
Position Encodingsconcept0.819
Mechanism for encoding sequence order in transformers; paper argues these should reflect learned structural representations rather than fixed sines/cosines.
what is the analogue of spatial positional encodings for higher order tasks such as language?question0.812
Open question raised in Discussion about extending TEM-t principles beyond spatial navigation.
Recurrent Position Encodingsmethod0.789
Key modification to transformers proposed in this paper: position encodings generated by a recurrent network trained on action sequences.
probably helps not only with faithful reconstruction but also creates interference patterns that encode nuanced information about the deltas and convergences between states.quote0.753
Key quote connecting path redundancy to interferometric information encoding.
Larger hidden representations create more random structure that DAS can search through, allowing manipulation of counterfactual behavior even in randomly initialized networkshypothesis0.750
Tested in Section 4.4 calibration experiment; confirmed by findings.
TEM-t learns band-cell-like position encoding representations resembling Krupic et al. band cellsfinding0.750
Empirical result showing TEM-t position encodings also recapitulate band cells, not just grid cells.
Smolensky (1986) proposes that viewing a neural representation under a basis that is not aligned with individual neurons can reveal the interpretable distributed structure of the neural representations.quote0.748
Load-bearing theoretical claim providing the conceptual foundation for DAS.