concept
active
concept:rotary-positional-embedding-ropeRotary Positional Embedding (RoPE)
A positional encoding mechanism that doesn't put positional information into the residual stream; changes which pointer arithmetic algorithms are available
Neighborhood — ranked by edge-count
Concepts (1)
concept
- Pointer Arithmetic in Transformersassociated_withA mechanism in models with positional embeddings where attention heads can manipulate and copy positional information to implement algorithms like induction heads
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Lagged time series used to capture dynamical dependencies.
- Technique where text is nested hierarchically within another, using indentation and margins to create subordinate orders of detail within an overarching embrace.
- The specific type of representation studied in the paper: function f: X→R^n assigning feature vectors to inputs
- A reinforcing interlock between different materials, mentioned alongside Deep Interlock in West Dean construction.
- Defines embedment as a spatially hierarchical technique.
- Prior work using steering vectors to control reflection, motivated by reducing redundant self-reflection in long CoT.
- Vector representations of individual tokens from genomic foundation models; the raw inputs to sequence pooling methods.
- Mechanism for encoding sequence order in transformers; paper argues these should reflect learned structural representations rather than fixed sines/cosines.