Stochastic text generation (next token prediction)

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Next Token Predictionconcept0.852
  The training objective of LLMs: predicting the most likely next token given context; formally P(w_{n+1}|w_1...w_n)
• Next-Token Prediction (NTP)method0.808
  Training objective used for all neural network models in the paper; cross-entropy loss over predicted token sequences.
• Autoregressive Text Generationconcept0.746
  LLMs generating text by feeding output back through context window; debated whether this constitutes algorithmic recurrence for RPT-1
• In Llama 3 70B, each next-token prediction at token 101 draws on 64,000 independent attention streams (8 heads × 80 layers × 100 prior positions), each carrying a 128-dimensional signalfinding0.737
  Quantitative argument for the richness of quasi-psychological connections enabled by attention streams
• stochastic parrotconcept0.733
  Derogatory term for LLMs; Nix's commentary frames it as camp opposite to nascent consciousness.
• Persona representations originate in pretraining, not alignment; they form without explicit supervision as useful features for next-token predictionclaim0.728
  Core interpretive claim providing mechanistic explanation for early persona formation
• We suspect it has to do with personas being latent factors that persist for many tokens; thus, recent expression of a persona should predict its near-future expression.quote0.728
  Load-bearing mechanistic conjecture about why persona vectors generalize from extraction to prediction
• A pair of query and key subcomponents distributed across attention heads performs previous-token behaviorfinding0.720
  VPD recovers an attention algorithm for attending to the previous token, distributed across multiple heads.