finding
active
finding:nla-misleads-agent-in-xqc-default-parameter-case-moving-0-56-to-0-32NLA misleads agent in xqc default-parameter case, moving 0.56 to 0.32
Qualitative loss.
Source paper
extracted_from(2026) · Adam Karvonen · Euan Ong · Subhash Kantamneni · Samuel Marks
Neighborhood — ranked by edge-count
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Soft clues redundant with transcript.
- Qualitative win.
- Tool-induced answer flips mostly toward 'no effect' (NLA 82% positive, 84% negative), cancelling in accuracyfinding0.720Direction of tool-driven prediction changes.
- Tool-discovered hypotheses are easily guessed.
- Shows a general code error detector beyond simple typo detection.
- Key limitation identified: NLAs hallucinate specific details while preserving thematic accuracy; informs practical usage.
- Explicit 'X causes Y' statement appears in only 0.6% (3 of 479) of positive claims' NLA outputsfinding0.708Tools rarely state causal relationships.
- Case study demonstrating NLA ability to surface root causes of model misbehavior; corroborated by training data inspection.