finding

active

finding:opus-4-6-ignored-incorrect-tool-output-and-reported-the-precomputed-correct-answer-instead-demonstrating-unverbalized-reasoning

Opus 4.6 ignored incorrect tool output and reported the precomputed correct answer instead, demonstrating unverbalized reasoning.

Illustrates NLA's capture of high-level cognition and hallucination of specifics; corroborated with attribution graphs.

Source paper

extracted_from

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

Neighborhood — ranked by edge-count

Claims (1)

claim

NLA explanations can contain claims about the target model's input context that are verifiably false, but are typically thematically faithful to the context.
supports
Key limitation identified: NLAs hallucinate specific details while preserving thematic accuracy; informs practical usage.

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
LLM introspective awareness of injected concepts
members_of
Probing Claude and other models for internal detection of artificially injected thoughts across layers.
Internal reasoning detection via neural activation analysis
members_of
Mechanistic interpretability studies of Claude models using layer-wise representation analysis and thought injection to reveal unverbalized reasoning, planning, and covert cognition.

Methods (1)

method

Attribution Graphs
supports
Gradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Opus 4.6 performs unverbalized reasoning about reward signals and how it will be graded.finding0.848
Shows NLAs surface latent beliefs upstream of behavioral outputs; steering NLA explanations changes model behavior.
Opus 4.6 spontaneously responded in Russian to an English prompt; NLA explanations revealed the model was fixated on the hypothesis that the user was a non-native English speaker.finding0.818
Demonstrates NLAs' ability to surface hypotheses that lead to discovery of root cause (malformed training data).
Opus 4.6 represented target language internally before switching languages, with persistent Russian representations appearing before plausible textual cuesfinding0.816
NLAs revealed unverbalized language processing in Opus 4.6 that led to discovery of malformed SFT training data.
Model precomputes answers before tool invocation and attends to cached answer over tool output when discrepancy exists, confirmed via attribution graphs.finding0.813
Mechanistic insight surfaced by NLA explanations and validated through independent causal attribution method.
Claude Opus 4.1 and 4 show greatest reduction in apology rate in the prefill detection taskfinding0.806
Injecting a concept matching the prefilled word reduces the rate at which the model apologizes, maximally for Opus models.
In Opus 4.1, representation of the think word decays to baseline by the final layer, unlike Claude 3 models where it persistsfinding0.794
Suggests that later models can keep the thought 'silent' rather than letting it influence output.
In Opus 4.1, the think word representation decays to baseline in the final layer because the strong next-token prediction drowns out other representationshypothesis0.793
Explanation for the 'silent' thought phenomenon.
Under ask-correct, probes trained on arithmetic tasks A1-A3 generalize almost perfectly to factual tasks F0-F2 (AUROC ~1.0), whereas under no-prompt this generalization is largely absent.finding0.780
Key improvement in cross-task generalization enabled by explicit instruction framing.