claim

active

claim:nla-explanations-can-contain-claims-about-the-target-model-s-input-context-that-are-verifiably-false-but-are-typically-thematically-faithful-to-the-context

NLA explanations can contain claims about the target model's input context that are verifiably false, but are typically thematically faithful to the context.

Key limitation identified: NLAs hallucinate specific details while preserving thematic accuracy; informs practical usage.

Source paper

extracted_from

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

Neighborhood — ranked by edge-count

Findings (1)

finding

Opus 4.6 ignored incorrect tool output and reported the precomputed correct answer instead, demonstrating unverbalized reasoning.
supports
Illustrates NLA's capture of high-level cognition and hallucination of specifics; corroborated with attribution graphs.

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
Natural Language Autoencoders for mechanistic interpretability
members_of
Using NLAs to extract human-readable explanations of model internals via unsupervised reconstruction, revealing steering vectors, confabulation patterns, and causal belief capture.
Natural Language Auditing of Neural Models
members_of
NLA explanations used as steering vectors and auditing tools to investigate model beliefs and misalignment.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

While NLA claims can be false in specifics, they are typically thematically faithful to contextclaim0.879
Key insight about confabulation patterns in NLAs enabling practical use.
NLA explanations confabulate false specifics but maintain thematic fidelity; claims repeated across tokens more likely true than isolated claims.claim0.858
Core limitation and usage heuristic: read NLAs for themes rather than individual factual claims; cross-check with original context.
Can NLAs provide mechanistic grounding of which aspects of an activation drove components of explanations?question0.800
Identified as a key limitation: NLAs are blackboxes by construction.
Automated auditing benchmark requiring end-to-end investigation of intentionally-misaligned model; NLA-equipped agents outperform baselines.finding0.797
Downstream task validating NLA utility for model auditing; agents succeed without access to misalignment training data.
A single direction encoding whether the model represents a claim as true or false is persona-relative across conversations (Lampinen et al. 2026)finding0.786
Additional evidence that core representations are persona-relative, supporting Claim about persona-relative representations
NLA-derived steering vectors from edited explanations can causally shift planning representations, changing rhyme completion from 'rabbit' to 'mouse' at ~50% success rate.finding0.782
Evidence that NLA explanations bear causal relationship to model outputs; demonstrates validity of extracted representations.
LLMs linearly represent truth-relevant information beyond the plausibility of text, as evidenced by probes trained on likely performing poorly on anti-correlated datasetsclaim0.782
Establishes that the observed linear structure is not merely a representation of text probability
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (Li et al., 2023)concept0.781
Safety intervention that relies on activation modification, which ESR might undermine