paper
active
2026
paper:doi-10-48550-arxiv-2608-16747

Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

TL;DR

Activation-reading interpretability tools give no uplift for predicting counterfactual model behavior on naturally occurring behaviors, even though the same tool families help in narrow fine-tuned-quirk auditing games. CHIVE (Counterfactual Hypothesis Investigation Via Edits) is an agentic pipeline that samples a target model 30 times per prompt on WildChat, screens for unexpected behavior with Opus 4.6, runs 5–15 prompt-edit experiments, and verifies the result with an independent judge. At about $1.9 per investigation, it yields thousands of explanations whose evaluation labels come only from measured resampling effects (true claims move behavior at least 50pp, false at most 15pp). On Gemma-3-27B-IT, activation oracles, natural-language autoencoders and SAEs leave an Opus 4.8 predictor at its transcript-only AUROC of about 0.81. This holds for Qwen3-8B, for GPT-5.5 and Gemini-3.1-Pro predictors, and across hyperparameter sweeps. NLA outputs state an explicit X-causes-Y relation in only 0.6% of positive claims. Training models to answer these claims as follow-up turns generalizes to held-out investigations, to PETRI out-of-distribution transcripts, and to hint settings. Qwen3-8B and Qwen3.5-397B land within ±0.03 AUROC of an Opus 4.8 reference, with no privileged-access advantage for self-training. The implication is that natural behaviors have context-specific causes that activations do not expose as anomalous content, so interpretability tools need evaluation on realistic counterfactual ground truth, and counterfactual investigation can supply both training and evaluation data at scale.

What to take away

  1. 1. On Gemma-3-27B-IT, an Opus 4.8 predictor with activation oracles, NLAs or SAEs scores AUROC around 0.81, the same as the transcript-only baseline of 0.814, so none of the three tools adds uplift.
  2. 2. The null result replicates on Qwen3-8B and with GPT-5.5 (0.819 baseline vs 0.809–0.812 with tools) and Gemini-3.1-Pro (0.793 baseline, dropping to 0.654 with the NLA and 0.746 with the SAE).
  3. 3. An LLM judge finds an explicit 'X causes Y' statement in only 3 of 479 positive claims (0.6%) of NLA outputs, so predictors must infer causality from correlational cues.
  4. 4. Tool outputs push predictions toward 'no effect' (82% of NLA flips on positive claims), which cuts accuracy on positive claims from 73% to 63% and raises it on negative claims from 74% to 81%, so the two effects roughly cancel.
  5. 5. Training on CHIVE counterfactual claims (one epoch, LoRA rank 64) lifts Qwen3-8B and Qwen3.5-397B-A17B to within ±0.03 AUROC of the Opus 4.8 reference on held-out WildChat and out-of-distribution PETRI investigations.
  6. 6. Training that never targeted hints transfers to hint settings (AITA sycophancy and MMLU cues), lifting Qwen3-8B's false-positive rate from 1.00 to 0.09 on sycophancy, where the base model always answered Yes.
  7. 7. Self-trained and cross-trained predictors (Qwen3-8B and Llama-3.1-8B, 23,944 matched examples) perform equally, so there is no evidence of privileged access, which contrasts with Li et al. (2026).
  8. 8. Replicable method: label claims by resampling 30 responses per edit and a frozen classifier question, calling a claim true at a shift of at least 50pp and false at no more than 15pp, with a 30pp threshold in the claim text.
  9. 9. Opus 4.1 and Opus 4.5 name the cause behind the four NLA-paper case studies within their top three guesses at least 60% of the time from a behavior description alone, which suggests hypothesis discrimination matters more than hypothesis generation.
  10. 10. Open question: whether the null stems from activations lacking causal relations (needing weight- or circuit-level access) or from tools and elicitation that could improve with training on CHIVE data.

Peer brief — for seminar discussion

CHIVE is an agentic pipeline for building ground-truthed explanations of naturally occurring LLM behavior. It samples a target model 30 times on WildChat prompts, has Opus 4.6 flag unexpected behaviors, runs 5–15 counterfactual prompt edits per behavior, and has a judge verify the result. Each investigation costs roughly $1.9. Labels derive only from resampled behavior rates: claims are true when an edit shifts the rate at least 50pp and false at 15pp or less, after LLM-judge filters for mechanism concreteness, reproducibility and confounds. That leaves 1,294 filtered Gemma-3-27B-IT claims from 4,433. The load-bearing finding is a null. Activation oracles, natural-language autoencoders and SAEs give an Opus 4.8 predictor no uplift over the transcript-only baseline (AUROC about 0.81). Table 4's sweeps over tool calls and token positions all straddle zero, and GPT-5.5 and Gemini-3.1-Pro show the same pattern, with Gemini actively hurt by the NLA (0.654 vs 0.793). The diagnostic is that NLA outputs state the causal link in only 0.6% of positive claims. The paper's account is that fine-tuned quirks appear as anomalous concepts absent from the prompt, whereas natural behaviors turn on a specific contextual feature. The second application is training. Models trained on one epoch of claims predict their own counterfactual outcomes on held-out WildChat and PETRI investigations and on hint settings. They land within ±0.03 AUROC of an Opus 4.8 reference, though a self-versus-cross-trained comparison finds no privileged access. Open-ended explanation training is mixed: it helps weak simulators and misleads strong ones. What it implies is that interpretability evaluation should move from planted quirks to realistic behaviors with behavioral ground truth. The authors hypothesize that either activations lack causal relations, or tools could improve if trained on this data. What is contestable is the proxy. Any evaluation answerable by running the counterfactual is only useful as a stand-in for cases without clean counterfactuals, and read-only access excludes the intervention methods, such as steering and activation patching, that interpretability often relies on. A critical reader would push hardest on this: the null shows that these three read-only tools, used by a prompted agent with five calls, don't beat a strong transcript reader on behaviors chosen because they have clean prompt-level causes. It does not show that interpretability fails on the hard cases that motivate it. The selection effect compounds this, since the pipeline keeps only behaviors with concrete mechanisms, and a transcript-only baseline on those is naturally strong. Training the tools on CHIVE data, which the authors suggest but do not run, would be the natural alternative test.

Methods (15)

Frameworks (2)

  • Auditing games
    Methodology measuring interpretability tool uplift for an agent uncovering hidden behaviors.
  • Counterfactual simulatability
    Lens for evaluating explanations by whether they help predict behavior on counterfactual inputs.

Datasets (6)

Findings (25)

Original abstract (expand)

Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+22 more

Similar preprints — Semantic Scholar