quote

active

quote:the-problem-isn-t-that-it-is-a-transformer-the-problem-is-that-it-is-an-auto-regressive-llm-auto-regressive-llms-that-compute-each-token-with-a-fixed-number-of-computational-steps-can-t-reason-regardless-of-the-details-of-the-architecture

The problem isn't that it is a transformer. The problem is that it is an auto-regressive LLM. Auto-regressive LLMs that compute each token with a fixed number of computational steps can't reason, regardless of the details of the architecture.

LeCun's post on X supporting the view that fixed-step probabilistic prediction precludes consciousness in LLMs.

Source paper

extracted_from

Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis

(2025) · Li, Jingkai

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

So at any point in the network, the transformer not only receives information from its past... but also has causal influence over its future processing. So, saying that LLMs cannot introspect... is incorrect.quote0.785
Core summary of Janus' position on autoregressive recurrence enabling introspection.
Transformers develop self-models through in-context learning, not just training data; even old base models without LLM-related text can bootstrap self-referential reasoning at runtime.claim0.782
Antra's foundational claim about how introspection arises computationally rather than from memorised text.
Transformers are recurrent through autoregression because K/V stream provides horizontal information flow across positions.claim0.771
Claim formalizing the Anima Labs idea that transformers are effectively recurrent due to K/V stream.
Sequences of contemporary Transformer-based LLM representations lack statistically significant indicators of observed 'consciousness' phenomena under the three stringent criteria.claim0.771
Primary conclusion of the study based on temporal permutation analysis failing all three criteria.
does the transformer genuinely use a local code for token-in-context features, or is dictionary learning producing a local code artifact from a compositional underlying representation?question0.769
Open question about the nature of the abundant token-in-context features found
The transformer likely uses a local code for token-in-context features rather than purely compositional representations, because local codes enable sharper predictionsclaim0.768
Authors argue the prevalence of token-in-context features reflects genuine model computation rather than dictionary learning artifact
Transformer-based LLMs do not clearly satisfy the GWT indicator properties.claim0.764
Assessment from case study.
Transformer can be viewed as a Wolfram causal graph with foliations specifying computation order.claim0.758
Janus's interpretive framing of transformers as causal graphs.