Can 'Consciousness' Be Observed from Large Language Model (LLM) Internal States? Dissecting LLM Representations Obtained from Theory of Mind Test with Integrated Information Theory and Span Representation Analysis

• David Chalmers

  cites
• Anil Seth

  cites
• Giulio Tononi

  cites
  Developer of integrated information theory; provides formal tools for measuring integration and consciousness in systems.
• Jason Wei

  cites
  Emergent abilities of LLMs.
• Thomas Nagel

  cites
  Philosopher referenced for 'what it's like' framework applied to understanding memory reconstruction from past self perspective.
• Douglas Hofstadter

  cites
• Bernard Baars

  cites
  Originator of Global Workspace Theory.
• Yoshua Bengio

  cites
• Ashish Vaswani

  cites
  Lead author of 'Attention is all you need', introducing the transformer architecture
• Larissa Albantakis

  cites
  Co-author with Hoel and Tononi on quantitative causal emergence.
• Blaise Agüera y Arcas

  cites
  AI researcher, co-author of articles on artificial general intelligence and evolving AI.
• David Premack

  cites
  Co-author of the classic Theory of Mind paper.
• Masafumi Oizumi

  cites
  First author of Oizumi, Albantakis, & Tononi (2014) 'From the phenomenology to the mechanisms of consciousness: integrated information theory 3.0'.
• Andrew Haun

  cites
  Applied IIT to ECoG data showing elevated Φ in conscious visual perception; key empirical precedent for IIT.
• Charlotte Caucheteux

  cites
  Author whose work on comparing LLM representations with human brain activity during NLP informed this study's layer sampling strategy.
• Ganesh Jawahar

  cites
  Author whose methodology for span representations from BERT is adopted in this study.
• Geoffrey Hinton

  cites
• Graham Findlay

  cites
  Co-author of a recent paper applying IIT to distinguish intelligence from consciousness in LLMs; argues feedforward LLMs yield low Φ.
• Isaac Nemirovsky

  cites
  Author of a key precedent study implementing IIT on resting-state fMRI data whose methodology this study closely adapts.
• James Strachan

  cites
  Lead author of the ToM test dataset used in this study.
• Jingkai Li

  authored
  Sole author of the paper; affiliated with OpenSci.World in Montréal.
• Martin Schrimpf

  cites
  Author whose findings suggest intermediate-to-deep LLM layers best predict human brain activity; guides layer selection in this study.
• Matjaz Gams

  cites
  Author who assessed ChatGPT consciousness using IIT alongside Turing Test, concluding low integration precludes consciousness.
• Matthew Peters

  cites
  Author of the span representation method (from ELMo paper) adopted in this study for computing Span Representations.

+4 more

• Integrated Information Theory

  implements
  Tononi et al. framework quantifying consciousness via integration; provides mathematical tools for measuring agent complexity.
• Representation Network (RN)

  introduces
  Novel construct introduced by this paper: a hypothetical graph embedded in the time series of LLM representations, where each dimension is a node and latent connections are edges.
• IIT 3.0

  implements
  Version 3.0 of IIT, used to compute Φmax and Conceptual Information (CI) from LLM representation networks.
• IIT 4.0

  implements
  Version 4.0 of IIT, used to compute Φ and Φ-structure from LLM representation networks; latest iteration at time of study.
• Span Representation Analysis

  implements
  Framework for characterizing span-level information of sequences of representations, independent of any consciousness estimate; used as a comparison baseline.
• Theory of Mind (ToM)

  implements
  The cognitive ability to attribute mental states to oneself and others; used as the empirical domain for testing LLM representations.

• Sequences of contemporary Transformer-based LLM representations lack statistically significant indicators of observed 'consciousness' phenomena under the three stringent criteria.

  introduces
  Primary conclusion of the study based on temporal permutation analysis failing all three criteria.
• Complement syntax and mental state verb comprehension abilities crucial for human ToM development are not significantly represented in LLMs, revealing fundamental discrepancies between natural and artificial intelligence regarding mind development.

  introduces
  Derived from the finding that linguistic span focusing on complements/MSV yields no significant IIT estimate changes.
• LLM representations exhibit intriguing patterns under spatio-permutational analyses, suggesting a potentially profound yet tentative indication of consciousness.

  introduces
  Qualified positive claim from spatio permutation analysis where two cases satisfy all three criteria.
• It is plausible that ongoing developments in LLMs may lead to models or agentic systems built on LLMs capable of generating representations observed with 'consciousness' phenomena.

  introduces
  Forward-looking claim suggesting the methodological framework is relevant for future AI systems beyond current LLMs.
• The LLM itself cannot 'experience' what it generates and therefore cannot possess consciousness; the RN is a higher-level construct that is independent of the LLM's architecture once representations are generated.

  introduces
  Key theoretical position distinguishing analysis of representations from analysis of LLM architecture.
• Theory of Mind is a subset of cognitive abilities enabled by consciousness, not its equivalent; consciousness is a prerequisite for ToM, but ToM is not the entirety of consciousness.

  introduces
  Theoretical clarification distinguishing ToM from consciousness to frame the study's approach.
• Variations in ToM test score categories are more likely attributed to span-level information of the LLM representation sequence rather than to a 'consciousness' phenomenon as suggested by IIT estimates.

  introduces
  Main interpretive finding from Criterion 3 comparison showing Span Representation consistently outperforms IIT under temporal permutation.
• PCA is the appropriate dimensionality reduction technique for constructing the RN because it preserves global structure and provides deterministic, interpretable projections.

  introduces
  Justifies PCA choice over UMAP or t-SNE for the node-structured RN model.
• Scaled dot-product attention is the most faithful, structured, and theoretically grounded method for incorporating stimulus influence into response representations leading to an RN.

  introduces
  Justifies the methodological choice of attention over concatenation, mean pooling, residual connections, or joint embedding.

• Attended Response Representations (ARR)

  introduces
  Time series of response representations contextualized by applying dot-product attention to the corresponding stimulus representations.
• Contextually Attended Response Representations (CARR)

  introduces
  Extension of ARR where attention is directed specifically to linguistic spans (complement syntax or mental state verbs) within the stimulus.

• We hypothesize that 'consciousness' phenomena can be observed in the internal states of an LLM, specifically in its learned representations when analyzed as a sequence.

  introduces
  Primary research hypothesis driving the entire study; operationalized via three criteria.
• We hypothesize that potential 'consciousness' phenomena are preferentially associated with deeper transformer layers and the 2/3 layer of LLMs.

  introduces
  Derived from observed alignment of promising cases with semantically rich deeper layers and the brain-aligned 2/3 layer.
• If 'consciousness' phenomenon can be observed from ToM-related RN, higher ToM test scores should yield higher values of μΦmax (IIT 3.0) and/or μΦ (IIT 4.0).

  introduces
  Specific prediction linking IIT's prediction of high Φ for good performance to the experimental design's scoring structure.
• We hypothesize that a Representation Network (RN) emerges from LLM representations, where each dimension is a node and latent connections exist between nodes or clusters of nodes.

  introduces
  Core methodological hypothesis enabling the application of IIT to LLM representation sequences.

• 2/3 Layer of LLM

  mentions
  The layer approximately two-thirds through an LLM's transformer stack, reported to best predict human brain activity; identified as promising for consciousness indicators.
• IIT Five Axioms (Existence, Composition, Information, Integration, Exclusion)

  mentions
  Foundational axioms of IIT from which postulates about physical systems are derived; applied to the RN in this study.
• Mixture-of-Experts (MoE)

  mentions
  Architecture of Mixtral-8x7B; uses sparse expert routing affecting how hidden states are computed across layers.
• Black-Box Nature of Deep Learning

  mentions
  The opacity of LLM internal representations that motivates this study's investigation of whether consciousness can be observed from them.

• Can 'consciousness' be observed in the internal states of an LLM, specifically in its learned representations, particularly when analyzed as a sequence?

  introduces
  The primary research question framing the entire study.
• Is 'experience' encoded in sequences of LLM representations beyond mere 'knowledge,' 'understanding,' 'value,' or 'position'?

  introduces
  Secondary question motivating the IIT analysis; asks whether LLM hidden states contain something beyond propositional content.
• What is the relationship between different dimensions or clusters of dimensions in LLM representations? Do they and/or how do they interact with each other?

  introduces
  Motivates the RN hypothesis by pointing to the unknown relational structure within high-dimensional representation vectors.

• Strachan et al. 2024 ToM Test Dataset (osf.io/dbn92)

  cites
  Publicly accessible dataset of ToM test results from human and LLM participants across five tasks; primary data source for this study.

• Natural Language Processing (Elsevier journal)

  cites
  Venue where this paper was published as a journal article.

• Supplementary code and data at doi.org/10.1016/j.nlp.2025.100163

  cites
  Code, labeled linguistic spans, and augmented responses made publicly available as supplementary material.