Autoregressive models

Second model system studied; used to show why flat autoregressive LLMs struggle with long-range coherence.

Neighborhood — ranked by edge-count

paper

method

Free-energy scaling under domain-wall formation
implements
Key analytical technique used across three model systems to determine constraints on long-range order.

framework

transformer architecture
extends
Neural network architecture based on attention, commonly used in large language models
Topological constraints on self-organisation framework
cites
Main framework: uses scaling of free energy under domain wall formation to determine whether local interactions can sustain ordered phases based on graph topology alone
Windowed Hamiltonian
implements
Core technical concept: a Hamiltonian where spin-spin interactions are defined only within finite windows ω, enabling generic analysis across diverse lattices

finding

Autoregressive model unable to converge to a single stored pattern for any finite β (Corollary 2)
supports
Consequence of Theorem 3 and 1D no-order result
Causally-masked attention in a decoder-only model has no ordered phase (Proposition 2)
about
Application to transformer language models
Rudimentary language models are challenged by long sequences of outputs.
cites
Empirical observation explained by topological constraints: flat autoregressive architectures lack multiscale structure needed for long-range order.

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

autoregressive modelingmethod0.949
Statistical technique where outputs are regressed on previous values; used in language generation
Autoregressive Language Modelingconcept0.869
Training objective interpretable as optimizing a diverse set of tasks; thus subject to multitask scaling convergence pressures
Autoregressive Samplingmethod0.864
The mechanism by which LLMs generate text: drawing a token from the next-token distribution and appending it to context repeatedly
Autoregressive Text Generationconcept0.838
LLMs generating text by feeding output back through context window; debated whether this constitutes algorithmic recurrence for RPT-1
autoregressive persistenceconcept0.828
Baseline persistence of any probe direction arising from the autoregressive nature of LLMs, not specific to emotion content
autoregressive recurrenceconcept0.822
Transformers are recurrent through autoregression because the K/V stream provides horizontal information flow across positions, even though each forward pass is feedforward.
autoregressive parallelizationconcept0.805
The training parallelization technique that latent methods are difficult to train with.
Associative memory modelframework0.792