finding
active
finding:tool-induced-answer-flips-mostly-toward-no-effect-nla-82-positive-84-negative-cancelling-in-accuracyTool-induced answer flips mostly toward 'no effect' (NLA 82% positive, 84% negative), cancelling in accuracy
Direction of tool-driven prediction changes.
Source paper
extracted_from(2026) · Adam Karvonen · Euan Ong · Subhash Kantamneni · Samuel Marks
Neighborhood — ranked by edge-count
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Soft clues redundant with transcript.
- Shows NLA explanations capture latent model beliefs about rewards before output selection; validates interpretability.
- Perturbation validation.
- Explicit 'X causes Y' statement appears in only 0.6% (3 of 479) of positive claims' NLA outputsfinding0.770Tools rarely state causal relationships.
- Key limitation identified: NLAs hallucinate specific details while preserving thematic accuracy; informs practical usage.
- Illustrates NLA's capture of high-level cognition and hallucination of specifics; corroborated with attribution graphs.
- Tool-discovered hypotheses are easily guessed.
- Evidence that NLA explanations bear causal relationship to model outputs; demonstrates validity of extracted representations.