finding

active

finding:autoregressive-model-unable-to-converge-to-a-single-stored-pattern-for-any-finite-corollary-2

Autoregressive model unable to converge to a single stored pattern for any finite β (Corollary 2)

Consequence of Theorem 3 and 1D no-order result

Source paper

extracted_from

Topological constraints on self-organisation in locally interacting systems

(2025) · Francesco Sacco · Dalton A R Sakthivadivel · Michael Levin

Neighborhood — ranked by edge-count

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
Autoregressive models and context window limitations
members_of
Theoretical and empirical analysis of why AR language models cannot maintain coherence or convergence beyond their context window through local interactions alone.
Autoregressive LLMs & formal thought disorder
members_of
Statistical physics arguments link LLMs' inability to maintain long-range coherence to schizophrenic derailment.

Frameworks (3)

framework

Autoregressive models
supports
Second model system studied; used to show why flat autoregressive LLMs struggle with long-range coherence.
Potts model
about
One of three model systems studied to analyze free-energy scaling and domain-wall formation in self-organizing systems.
AR(ω) model
about
Stochastic process model predicting next token from a context window of length ω; mapped to local Hamiltonian

Findings (2)

finding

For one-dimensional local Hamiltonian with m>1 stored patterns at non-zero temperature, domain wall formation is thermodynamically favourable (Theorem 2)
supports
No ordered phase in 1D with multiple stored patterns
A unique local Hamiltonian with window length ω can be associated to any AR(ω) model (Theorem 3)
supports
Mapping autoregressive models to spin systems

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Autoregressive language models cannot converge to single stored patterns beyond their context window from local interactions alone.claim0.863
autoregressive persistenceconcept0.787
Baseline persistence of any probe direction arising from the autoregressive nature of LLMs, not specific to emotion content
autoregressive modelingmethod0.775
Statistical technique where outputs are regressed on previous values; used in language generation
Different models cannot converge to the same representation if they have access to fundamentally different information; convergence is capped by mutual information between input signalsclaim0.768
Key limitation of the PRH for non-bijective observations
Purely local autoregressive systems cannot maintain long-range coherence at finite temperature.claim0.767
Any system that persists must minimise surprisal, thereby gathering evidence for its own generative model.quote0.764
Opening sentence defining self-evidencing.
The inability for autoregressive large language models to maintain states of long-range order resembles tangential speech or derailment in formal thought disorder.claim0.762
Analogy between LLM incoherence and schizophrenia symptoms
If loss keeps going down on the test set, in the limit the model must be learning to interpret and predict all patterns represented in language, including common-sense reasoning, goal-directed optimization, and deployment of the sum of recorded human knowledge.hypothesis0.760
Extrapolation of scaling predictive models to AGI.