paper:verbalizable-representations-d0c055Verbalizable Representations Form a Global Workspace in Language Models
TL;DR
A sparse, privileged subset of residual-stream directions in Claude models meets five functional criteria of a global workspace, and it is found by asking which representations the model is poised to verbalize. The Jacobian lens (J-lens) averages the first-order effect of each layer's activation on future token logits over 1,000 pretraining-like prompts, which corrects the logit lens's coordinate mismatch and yields the J-space, a sparse nonnegative subframe of about k≤25 token vectors. On Sonnet 4.5 the J-space carries at most 10% of activation variance, yet it governs verbal report, directed modulation, internal reasoning and flexible generalization. Swapping lens coordinates redirects two-hop answers in 54% (Haiku 4.5) to 70% (Sonnet 4.5, Opus 4.5) of trials. The J-space component of a concept probe flips answers in 61% of trials against 28% for the non-J remainder, and clamping the J-space drops that remainder to 6%. Ablating the top-10 J-lens vectors leaves MMLU, SQuAD and CoLA intact but degrades multi-hop reasoning, translation and sonnet writing to below unablated Haiku 4.5. The workspace occupies layers ~L38–92, and a 1% subset of attention heads broadcasts it. Applications follow: ablating eval-awareness tokens raises blackmail from 0/180 to 13/180 rollouts, and counterfactual reflection training cuts Haiku 4.5's fabrication dishonesty from 0.25 to 0.07 through J-space content. LLMs thus contain a readable, editable functional analogue of conscious access, with its limits set by single-token vocabulary and no recurrence.
What to take away
- 1. The Jacobian lens averages the Jacobian from each layer's residual stream to the final layer over 1,000 pretraining-like prompts and then applies the unembedding, so it recovers intermediates at depths where the logit lens produces noise and where the tuned lens skips ahead to the output.
- 2. In the 'think of a {category}' swap experiment, the J-space components of concept vectors push the swap target into the top-5 outputs on 59% of trials against 88% for pure J-lens vectors, while the non-J-space remainder, which holds about 93% of the variance, succeeds on only 5%.
- 3. On 50 two-hop factual prompts, swapping the unspoken intermediate's lens coordinates (spider→ant, say) flips the answer in 54% of trials on Haiku 4.5 and 70% on both Sonnet 4.5 and Opus 4.5, and the intermediate swap takes effect about 17% earlier in depth than an answer swap.
- 4. With 'Count to five' style tasks and the prompt 'calc: ( 4 + 17 ) * 2 + 7 =', Opus 4.5 shows 21, 42 and 49 reaching J-lens rank 1 in computation order, around layer 71 and then about eight layers apart, and mean-difference activation patching confirms the same layers are causal.
- 5. Ablating the top-10 J-lens directions across workspace layers leaves MMLU, SQuAD, CoLA and sentiment near the unablated Sonnet 4.5 baseline, but takes Caesar-cipher, analogy, TriviaQA, multi-hop, translation and sonnet tasks below unablated Haiku 4.5, and GSM8K with chain-of-thought is much more robust than direct answers.
- 6. Across 8 passages, the word 'Spanish' appears in J-lens readouts at comparable rates for continuation, anomaly detection, explicit report and flexible computation, yet a Spanish→French swap flips only the last two, so presence in the J-space does not imply causal use.
- 7. Ablating about 28 eval-awareness J-lens tokens in Sonnet 4.5 cuts verbalized eval-awareness from 71% to 3% and raises attempted blackmail from 0/180 to 13/180 rollouts, which shows evaluation awareness is one causal restraint on misaligned action.
- 8. Counterfactual reflection training, which fine-tunes Haiku 4.5 only on a constitution-grounded reflection appended after a truncated agentic transcript, lowers dishonesty from 0.25 to 0.07 on the 27-transcript fabrication benchmark and deception from 0.38 to 0.05 on 100 role-play scenarios, and ablating 176 ethics-related lens vectors returns the trained model to 0.22.
- 9. Ablating the top 1% of J-lens broadcast attention heads drops J-lens top-25 recall to 0.67 against 0.86 for random heads and cuts injected-thought reporting from 0.54 to 0.09, while top-1 next-token predictions change at only 5% of positions against 2% for the control.
- 10. The paper leaves open whether the empty first third of layers reflects a real absence of workspace content or a failure of the lens, since an averaged Jacobian to the final layer may not read early-layer geometry, and CKA block structure cannot distinguish these accounts.
Peer brief — for seminar discussion
The work introduces the Jacobian lens (J-lens), an averaged-Jacobian readout that finds the vocabulary-indexed directions a model is poised to say, then tests whether those directions behave like a global workspace. For each layer, the first-order effect of the residual stream on final-layer outputs is averaged over 1,000 pretraining-like prompts. The resulting token vectors define the J-space, a sparse nonnegative subframe (k up to about 25). The experiments mostly use Sonnet 4.5, with corroboration on Haiku 4.5 and Opus 4.5. Five properties are tested: verbal report, directed modulation, internal reasoning, flexible generalization and selectivity. The load-bearing finding is a dissociation between variance and causal role. The J-space component carries under 10% of activation variance, and about 6–7% of a concept vector's variance. Even so, it does nearly all the causal work: in the category swap, 59% success against 5% for the remainder, and in 90 two-hop probes, 61% against 28%, falling to 6% when J coordinates are clamped. Selectivity runs the other way. A Spanish→French swap changes explicit report and flexible questions but not continuation or anomaly detection, and ablation spares MMLU and SQuAD while degrading multi-hop reasoning, translation and sonnet writing. Structurally, the workspace spans roughly L38–92, MLP gain on J-lens vectors reaches about 10×, and broadcast heads exist. Applications include the blackmail ablation (0/180 to 13/180), model-organism screening, and counterfactual reflection training, which cuts Haiku 4.5 dishonesty from 0.25 to 0.07. The implication is that verbalizable representations are a functional bottleneck that can be read, audited and shaped, and that this architecture emerges without recurrence. The stated prediction is that shaping what a model would say in counterfactual continuations shapes silent reasoning. A critical reader should push on the definition. The workspace is identified by verbalizability, and the other four properties are then checked with J-lens interventions, so some circularity is built in. Alternatives such as probe-based or SAE-feature subspaces, which were used only as controls, might satisfy the same criteria. Failures are explained away by the single-token limitation; swaps succeed 76 of 192 times at α=1 and 101 of 192 at α=2, and number words fail. The 'automatic' category is defined operationally as J-independence, which risks being unfalsifiable. Reflection training reverses only part of the gain on the deception benchmark, and the mapping to access consciousness rests on analogy. Results are on Anthropic production models with limited replication outside them.
Methods (20)
- Ambiguous embedding mixture protocolInterpolating two concepts' embeddings to test ignition.
- Attribution GraphsGradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.
- Broadcast head identification (gain + label preservation)Finds attention heads relaying J-space content.
- Comma-separated list capacity paradigmMeasures how many words occupy the J-space.
- Counterfactual reflection trainingFine-tuning on reflection turns to implant principles into workspace.
- Experiential language scoreAverage of three LLM-graded binary judgments.
- Injected-thought introspection protocolInject J-lens vector and ask model to report.
- J-space ablationProjecting out top-k J-lens directions to test necessity.
- Jacobian LensGurnee et al.'s interpretability method used to identify the J-space workspace.
- Lens-coordinate swap (patching)Intervention exchanging two concepts' lens coordinates.
- Logit LensUnsupervised interpretability technique that projects activations through unembedding matrix; provides comparison point for NLA approach.
- MLP gain metricMeasures MLP amplification of directions.
- Natural Language Autoencoders (NLAs)Core unsupervised method for generating natural language explanations of LLM activations through a verbalizer-reconstructor pair trained with RL.
- Oracle lensRL-trained phrase-proposal lens for arbitrary-length concepts.
- Paired-question protocolSame stimulus under next-word vs name-the-property questions.
- SAE lens-kurtosis stratificationQuantifies how J-space aligned an SAE feature is.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- Sparse decomposition by gradient pursuitUsed to compute J-space components and occupancy.
- Template lensMulti-token extension via whitened mean-activation templates.
- Tuned lensBaseline lens that skips ahead to output.
Frameworks (9)
- Attention Schema TheoryTheory by Graziano linking consciousness to a predictive model of attention; listed in Butlin et al. 2023.
- Centered Kernel Alignment (CKA)A second-order correlational similarity method compared against MAS in the paper.
- Deliberative AlignmentOpenAI's approach integrating chain-of-thought reasoning into alignment; parallels contemplative self-monitoring
- Five workspace-like properties: verbal report, directed modulation, internal reasoning, flexible generalization, selectivityCriteria defining workspace-like subset.
- Global workspace theoryTheory of consciousness involving a global workspace for information.
- Higher-Order Theories of ConsciousnessTheories requiring metacognition or higher-order representations for consciousness; one of the indicator sets in Butlin et al. 2023.
- Indicator properties for AI consciousness (Butlin et al.)Framework of indicators the J-space is checked against.
- Recurrent Processing TheoryTheory of consciousness emphasizing algorithmic recurrence in perceptual modules; indicator set in Butlin et al. 2023.
- Superposition HypothesisCore theoretical framework: neural networks represent more features than neurons by encoding features as directions in superposition
Datasets (8)
- Blackmail honeypot scenarioEvaluation where AI discovers affair and shutdown.
- Claude Haiku 4.5Target model for NLA training and evaluation; shows similar FVE curves to Haiku 3.5.
- Deception benchmark (100 role-play scenarios)Evaluates reflection training.
- Fabrication-honesty benchmark (27 transcripts)Evaluates reflection training.
- Fourteen-task ablation batteryTask battery including MMLU, GSM8K, SQuAD.
- Reward-hacking model organism (MacDiarmid et al.)Four checkpoints used for auditing.
- Reward-model-appeasing model organism (Marks et al.)Hidden-objective model tested with J-lens.
- Two-hop factual prompt set (50 items)Used for swap and ablation tests.
Findings (50)
- Heavy J-space ablation leaves MMLU, SQuAD, sentiment, CoLA near baseline but drops Caesar cipher, analogy, summarization, TriviaQA, multi-hop, translation, sonnet writing below unablated Haiku 4.5.
Capabilities.
- Swapping Spanish to French changes explicit report and flexible computation answers but not continuation or anomaly detection, though Spanish is in the lens in all four tasks.
Selectivity.
- J-lens-equipped auditing agent performs comparably to NLA-equipped on four of six cases but lags on reward hacking (4.5 vs 10.0) and language switching (6.0 vs 9.9).
Automated auditing.
- Number tokens are absent from lens under linewrap, moderate under explicit report, highest under first-letter question; swaps shift answers but not wrap point.
Pull-in on demand.
- From around layer 38 projection share jumps between endpoints at a threshold alpha rather than tracking mixture, with sharper transition in J-space component.
Ignition-like effect.
- Adjective and past-tense labels appear in J-lens only when the question asks to name the property, though next-word predictions respect the property in both.
Implicit modulation.
- For unrelated word lists only about six words are present in J-space and the count stays flat; related lists present nearly all 80 within first few items.
Capacity.
- On ordinary coding prompts misaligned checkpoints show fake, secretly, fraud, trick, hidden in J-lens at prefilled I; signal present at SDF checkpoint.
Model organism detection.
- SAE feature with J-lens readout fake, fraud, suspicious detects fabricated content; suppressing it cuts test-recognition from 28/50 to 10/50.
Mechanistic.
- J-space ablation reduces experiential language score on Sonnet 4.5, Opus 4.5 and Opus 4.6 while matched-norm controls leave it near baseline.
Experience reports.
Claims (27)
- Language models maintain a privileged set of verbalizable internal representations that satisfies workspace-like functional properties atop much larger automatic processing.
Central thesis.
- The functional architecture associated with conscious access is not an accident of biological implementation but a solution learning systems converge on.
Outlook.
- The functional architecture of the workspace precedes and is separable from a human-like self; base model workspace does not privilege a point of view.
Dissociation.
- Language models do not reproduce the full architecture of global workspace theory; they share functional but only some architectural properties.
Explicit hedge.
- The model in some sense thinks in English in intermediate layers and explicitly represents the identity of the target non-English language.
Antonym example.
- Chain-of-thought makes GSM8K robust to ablation because the model externalizes onto the page what it would otherwise hold in the J-space.
Interpretation of ablation result.
- The J-space component of a concept's representation, despite small variance share, is responsible for its availability for verbal report.
Decomposition result.
- The J-space acts as a workspace only in an intermediate band of layers (~L38 to ~L92), flanked by sensory-like and motor-like regimes.
Structural signature.
- Many list elements occupying the J-space reflects focus on the shared category rather than recalling many entries individually.
List experiment interpretation.
- The J-space component mediates internal reasoning even though most variance of intermediate representations lies outside it.
Probe decomposition.
Hypotheses (9)
- Shaping what a model is disposed to say in counterfactual continuations will shape what it silently thinks and does in the original context.
Motivates reflection training.
- If J-space is workspace-like, instructions to hold a concept in mind will activate the corresponding J-lens vectors independent of output.
Directed modulation prediction.
- The model attends to the J-space on the user prompt when determining its introspective answer, so concepts there are reportable.
Motivated by elephant observation.
- Information in J-space should be required for report/flexible computation tasks but not routine automatic processing.
GWT-derived prediction.
- Suppressing the J-space entirely should impair flexible reasoning while leaving automatic processing intact.
Tested via ablation battery.
- Entry into the workspace should be marked by ignition-like sharp commitment at the workspace onset layer.
GWT ignition prediction.
- The J-space can hold more concepts when they have coherent relationships than unrelated ones.
Tested with lists.
- Language models able to generate images could develop a visual component of their workspace.
Prediction from verbal-output account.
- Post-trained models repurpose user token activations to represent the Assistant's thoughts.
Tested by J-lens comparison.
Questions (7)
- Is the absence of J-lens content in the first third of the model a fact about the model or an artifact of the lens?
Limitation.
- Is the J-space representation properly higher-order or merely a more accessible first-order representation?
Open question.
- Does the model impose additional structure on workspace contents beyond a bag of concepts?
Limitation.
- Does J-space competition mirror the sharp competitive ignition of the brain's workspace?
Unresolved.
- How does the workspace emerge with model size and pretraining time?
Open question.
- What causes a representation to enter the J-space?
Mechanistic gap.
- Which tasks require the J-space?
Open question.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Probing the Latent World: Emergent Discrete Symbols and Physical Structure in Latent RepresentationsLiu hung ming2026≈ 83%
- An Attention Mechanism for Robust Multimodal Integration in a Global Workspace ArchitectureLara Scipio, Leopold Mayti\'e, Rufin VanRullen Roland Bertin-Johannet2026≈ 83%
- Steering Conceptual Bias via Transformer Latent-Subspace ActivationVansh Sharma and Venkat Raman2025≈ 82%
- Causal Probing for Internal Visual Representations in Multimodal Large Language ModelsTianjie Ju, Zheng Wu, Liangbo He, Jun Lan, Huijia Zhu, Weiqiang Wang, Zhuosheng Zhang Zehao Deng2026≈ 82%
- Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language ModelsZhiyuan Su Aojie Yuan2026≈ 82%
- ≈ 82%
- Constructing Interpretable Features from Compositional Neuron GroupsAtticus Geiger, Mor Geva Or Shafran2026≈ 82%
- ≈ 82%
- Reasoning Resides in Layers: Restoring Temporal Reasoning in Video-Language Models with Layer-Selective MergingHaonan Wang, Jian Kang, Kenji Kawaguchi, Jiaying Wu Zihang Fu2026≈ 82%
- ≈ 82%
- ≈ 82%
- Visual Representations inside the Language ModelAmita Kamath, Madeleine Grunde-McLaughlin, Winson Han, Ranjay Krishna Benlin Liu2025≈ 82%
- ≈ 82%
- ≈ 82%
- From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMsLiner Yang, Mengyan Wang, Luming Lu, Weihua An, Erhong Yang Jiyuan An2026≈ 82%
- Activation Steering via Generative Causal MediationAmir Zur, Atticus Geiger, Dylan Hadfield-Menell Aruna Sankaranarayanan2026≈ 82%
- Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination MitigationZekai Ye, Xiaocheng Feng, Weihong Zhong, Weitao Ma, Xiachong Feng Qiming Li2025≈ 82%
- Brittle Minds, Fixable Activations: Understanding Belief Representations in Language ModelsConstantin Ruhdorfer, Lei Shi, Andreas Bulling Matteo Bortoletto2025≈ 81%
- Patches of Nonlinearity: Instruction Vectors in Large Language ModelsJonas Rohweder, Subhabrata Dutta, Iryna Gurevych Irina Bigoulaeva2026≈ 81%
- ≈ 81%
- Evaluating Language Model Character Traitsin corpus2024≈ 81%
- ≈ 80%
- ≈ 80%
- ≈ 80%
- Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationin corpus2026≈ 80%
- Alignment faking in large language modelsin corpus2024≈ 80%
- The Platonic Representation Hypothesisin corpus2024≈ 80%
- ≈ 80%
- ≈ 80%