Position-Only Keys/Queries, Stimulus-Only Values Factorization

Key architectural modification restricting queries and keys to position encodings while values depend only on stimuli; extreme version of best-practice insight.

Neighborhood — ranked by edge-count

Papers (1)

paper

Relating transformers to models and neural representations of the hippocampal formation
introduces

Frameworks (1)

framework

TEM-Transformer (TEM-t)
implements
The transformer version directly analogous to TEM, introduced in this paper, offering dramatic performance improvements.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Position Encodingsconcept0.734
Mechanism for encoding sequence order in transformers; paper argues these should reflect learned structural representations rather than fixed sines/cosines.
Key-query attentionconcept0.732
Attention mechanism in AI used in Transformers; also proposed for GWT implementations.
Key, query, and value vectors are intermediary byproducts; W_OV and W_QK are the fundamental low-rank matrices describing attention head behaviorclaim0.724
Reframing observation: the canonical K/Q/V decomposition is computationally convenient but not the most interpretable representation
Decompose parameters, not activationsquote0.713
Core slogan encapsulating the paradigm shift of VPD.
reassociation of values upon key unreachabilityconcept0.706
When an intermediate key in a chain becomes unreachable, the model should re-associate values directly between the predecessor and successor.
Smith & Boyd (1991) criteria are irrelevant across the vast majority of the space of possible agents.claim0.704
Dismissal of earlier criteria as too narrow.
Keys as Meta-Taxonomic Dimensionsconcept0.701
Each key position n creates an independent taxonomic dimension; additional key positions enable multi-dimensional delegation and namespace constraints.
Mouse as Position-Dependent Input Deviceconcept0.701