claim

active

claim:decoder-only-transformer-architectures-are-fundamentally-limited-in-generating-long-coherent-sequences-due-to-lack-of-ordered-phase

Decoder-only transformer architectures are fundamentally limited in generating long, coherent sequences due to lack of ordered phase.

Interpretation of Proposition 2 as a fundamental limitation on LLMs

Source paper

extracted_from

Topological constraints on self-organisation in locally interacting systems

(2025) · Francesco Sacco · Dalton A R Sakthivadivel · Michael Levin

Neighborhood — ranked by edge-count

Findings (1)

finding

Causally-masked attention in a decoder-only model has no ordered phase (Proposition 2)
supports
Application to transformer language models

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
Mechanistic structure of transformer attention computations
members_of
Identifies distributed algorithms implemented across attention heads, with focus on causal masking limitations and emergent capabilities via activation manifold steering.
Causal masking phase transitions in transformers
members_of
Studies how decoder-only architectures lack ordered phases necessary for coherent long-sequence generation due to causally-masked attention constraints.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Transformer decoder architectureframework0.801
Base architecture of reasoning LLMs studied, with attention and MLP blocks per layer
Transformer architecture permits introspection; saying LLMs cannot introspect on past internal states is wrong.claim0.760
Janus's central claim that the architecture enables introspection, though usage in practice is a separate question.
The problem isn't that it is a transformer. The problem is that it is an auto-regressive LLM. Auto-regressive LLMs that compute each token with a fixed number of computational steps can't reason, regardless of the details of the architecture.quote0.756
LeCun's post on X supporting the view that fixed-step probabilistic prediction precludes consciousness in LLMs.
The transformer likely uses a local code for token-in-context features rather than purely compositional representations, because local codes enable sharper predictionsclaim0.748
Authors argue the prevalence of token-in-context features reflects genuine model computation rather than dictionary learning artifact
Transformer cognition operates via interference patterns across redundant paths, encoding nuanced state deltas similar to continuous human memory.claim0.744
Proposes transformers experience cognition as interference-based and continuous; connects to Anima Labs reports of parallel processing.
The last layer of the transformer has the largest projection magnitude on the reflection direction, likely because it directly controls generation of reflection keywordsclaim0.743
Interpretive claim from attention head attribution analysis in appendix
Different introspective tasks may preferentially use different path distributions in the transformer.claim0.742
Interpretive claim connecting exponential path combinatorics to Lindsey's layer-dependent findings.
Transformers are recurrent through autoregression because K/V stream provides horizontal information flow across positions.claim0.741
Claim formalizing the Anima Labs idea that transformers are effectively recurrent due to K/V stream.