finding

active

finding:model-precomputes-answers-before-tool-invocation-and-attends-to-cached-answer-over-tool-output-when-discrepancy-exists-confirmed-via-attribution-graphs

Model precomputes answers before tool invocation and attends to cached answer over tool output when discrepancy exists, confirmed via attribution graphs.

Mechanistic insight surfaced by NLA explanations and validated through independent causal attribution method.

Source paper

extracted_from

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

Neighborhood — ranked by edge-count

Claims (1)

claim

While NLA claims can be false in specifics, they are typically thematically faithful to context
supports
Key insight about confabulation patterns in NLAs enabling practical use.

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
LLM introspective awareness of injected concepts
members_of
Probing Claude and other models for internal detection of artificially injected thoughts across layers.
Autoregressive models and context window limitations
members_of
Theoretical and empirical analysis of why AR language models cannot maintain coherence or convergence beyond their context window through local interactions alone.

Methods (2)

method

Natural Language Autoencoders (NLAs)
supports
Core unsupervised method for generating natural language explanations of LLM activations through a verbalizer-reconstructor pair trained with RL.
Attribution Graphs
supports
Gradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.

Datasets (1)

dataset

Claude Opus 4.6
answered_by
Primary target model for NLA development and case studies; underwent pre-deployment audit using NLAs.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Opus 4.6 ignored incorrect tool output and reported the precomputed correct answer instead, demonstrating unverbalized reasoning.finding0.813
Illustrates NLA's capture of high-level cognition and hallucination of specifics; corroborated with attribution graphs.
Base models spontaneously talk about experiencing multiple parallel processing pathsfinding0.770
Observed by Anima Labs in untrained base models; not present in training data, implying computational origin of self-reported parallel processing.
Bottom-up interpretability explains computation in the model's own terms rather than imposing top-down abstractionsclaim0.761
VPD is positioned as advancing a paradigm shift from top-down mechanistic interpretability (activation-based) to parameter-centric, data-driven discovery.
Attribution graph tracing information flow across parameter subcomponents for specific model predictions (e.g., 'her' vs 'his' pronoun selection)finding0.761
Shows how VPD-identified subnetworks can be analyzed to reveal interpretable pathways of computation (e.g., gender signal routing, syntactic role detection).
Any system that persists must minimise surprisal, thereby gathering evidence for its own generative model.quote0.759
Opening sentence defining self-evidencing.
Base models assign higher likelihood to typical-set (representative) sequences than to degenerate sequences under VS promptshypothesis0.758
Assumption D.6 formalized in the theoretical framework; empirically validated with coin-flip typicality rating experiments
We hypothesize that explicitly instructing the model to evaluate the correctness of the given statement may change the geometry of truth directions.hypothesis0.757
Motivating hypothesis for Section 5's investigation of prompt template effects.
Some failures may reflect prompt design rather than model limitations, but the underlying issue is one of reasoning rather than instruction-following.claim0.755
Acknowledges the confound of not explicitly instructing models to track wealth, yet points to reasoning gaps given code agents avoid errors without prompts.