claim

active

claim:nla-explanations-confabulate-false-specifics-but-maintain-thematic-fidelity-claims-repeated-across-tokens-more-likely-true-than-isolated-claims

NLA explanations confabulate false specifics but maintain thematic fidelity; claims repeated across tokens more likely true than isolated claims.

Core limitation and usage heuristic: read NLAs for themes rather than individual factual claims; cross-check with original context.

Source paper

extracted_from

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

Neighborhood — ranked by edge-count

Findings (1)

finding

Little evidence of steganography between AV and AR; meaning-preserving transformations cause only small FVE drops.
contradicts
NLA explanations appear to encode information transparently in natural language rather than hidden channels.

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
Natural Language Autoencoders for mechanistic interpretability
members_of
Using NLAs to extract human-readable explanations of model internals via unsupervised reconstruction, revealing steering vectors, confabulation patterns, and causal belief capture.
Natural Language Auditing of Neural Models
members_of
NLA explanations used as steering vectors and auditing tools to investigate model beliefs and misalignment.

Methods (1)

method

Natural Language Autoencoders (NLAs)
contradicts
Core unsupervised method for generating natural language explanations of LLM activations through a verbalizer-reconstructor pair trained with RL.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

NLA explanations can contain claims about the target model's input context that are verifiably false, but are typically thematically faithful to the context.claim0.858
Key limitation identified: NLAs hallucinate specific details while preserving thematic accuracy; informs practical usage.
While NLA claims can be false in specifics, they are typically thematically faithful to contextclaim0.845
Key insight about confabulation patterns in NLAs enabling practical use.
Of the three categories of giving false information, only confabulation is directly applicable to LLM-based dialogue agentsclaim0.794
The paper distinguishes confabulation from good-faith error and deliberate deception, arguing the first is intrinsic to LLMs
NLA explanations grow more informative over training with FVE increasing from 0.3-0.4 to 0.6-0.8 roughly linearly in log(training steps)finding0.773
Quantitative evidence that NLA training produces increasingly informative explanations despite optimizing only for reconstruction.
Little evidence of steganography in NLAs; meaning-preserving transformations cause only small drops in FVEfinding0.764
Quantitative evaluation showing NLAs do not heavily rely on covert encoding beyond overt language.
Universality claims for truth directions are more limited than previously assumed, with significant differences observable for various model layers, task difficulties, task types, and prompt templates.claim0.763
Overarching conclusion summarizing the paper's contribution relative to prior universality claims.
Neutral NLI predictions may capture lexical rather than semantic diversityhypothesis0.762
Hypothesis proposed to explain Neutral NLI Diversity's high performance on decTest but low on conTest
Verbal reports, homology of structure or materials, phylogenetic provenance are all insufficient to make dependable conclusions across the space of possible agents.claim0.757
Summary assertion that traditional evidence fails for novel agents.