paper:doi-10-48550-arxiv-2608-16747Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments
TL;DR
Activation-reading interpretability tools give no uplift for predicting counterfactual model behavior on naturally occurring behaviors, even though the same tool families help in narrow fine-tuned-quirk auditing games. CHIVE (Counterfactual Hypothesis Investigation Via Edits) is an agentic pipeline that samples a target model 30 times per prompt on WildChat, screens for unexpected behavior with Opus 4.6, runs 5–15 prompt-edit experiments, and verifies the result with an independent judge. At about $1.9 per investigation, it yields thousands of explanations whose evaluation labels come only from measured resampling effects (true claims move behavior at least 50pp, false at most 15pp). On Gemma-3-27B-IT, activation oracles, natural-language autoencoders and SAEs leave an Opus 4.8 predictor at its transcript-only AUROC of about 0.81. This holds for Qwen3-8B, for GPT-5.5 and Gemini-3.1-Pro predictors, and across hyperparameter sweeps. NLA outputs state an explicit X-causes-Y relation in only 0.6% of positive claims. Training models to answer these claims as follow-up turns generalizes to held-out investigations, to PETRI out-of-distribution transcripts, and to hint settings. Qwen3-8B and Qwen3.5-397B land within ±0.03 AUROC of an Opus 4.8 reference, with no privileged-access advantage for self-training. The implication is that natural behaviors have context-specific causes that activations do not expose as anomalous content, so interpretability tools need evaluation on realistic counterfactual ground truth, and counterfactual investigation can supply both training and evaluation data at scale.
What to take away
- 1. On Gemma-3-27B-IT, an Opus 4.8 predictor with activation oracles, NLAs or SAEs scores AUROC around 0.81, the same as the transcript-only baseline of 0.814, so none of the three tools adds uplift.
- 2. The null result replicates on Qwen3-8B and with GPT-5.5 (0.819 baseline vs 0.809–0.812 with tools) and Gemini-3.1-Pro (0.793 baseline, dropping to 0.654 with the NLA and 0.746 with the SAE).
- 3. An LLM judge finds an explicit 'X causes Y' statement in only 3 of 479 positive claims (0.6%) of NLA outputs, so predictors must infer causality from correlational cues.
- 4. Tool outputs push predictions toward 'no effect' (82% of NLA flips on positive claims), which cuts accuracy on positive claims from 73% to 63% and raises it on negative claims from 74% to 81%, so the two effects roughly cancel.
- 5. Training on CHIVE counterfactual claims (one epoch, LoRA rank 64) lifts Qwen3-8B and Qwen3.5-397B-A17B to within ±0.03 AUROC of the Opus 4.8 reference on held-out WildChat and out-of-distribution PETRI investigations.
- 6. Training that never targeted hints transfers to hint settings (AITA sycophancy and MMLU cues), lifting Qwen3-8B's false-positive rate from 1.00 to 0.09 on sycophancy, where the base model always answered Yes.
- 7. Self-trained and cross-trained predictors (Qwen3-8B and Llama-3.1-8B, 23,944 matched examples) perform equally, so there is no evidence of privileged access, which contrasts with Li et al. (2026).
- 8. Replicable method: label claims by resampling 30 responses per edit and a frozen classifier question, calling a claim true at a shift of at least 50pp and false at no more than 15pp, with a 30pp threshold in the claim text.
- 9. Opus 4.1 and Opus 4.5 name the cause behind the four NLA-paper case studies within their top three guesses at least 60% of the time from a behavior description alone, which suggests hypothesis discrimination matters more than hypothesis generation.
- 10. Open question: whether the null stems from activations lacking causal relations (needing weight- or circuit-level access) or from tools and elicitation that could improve with training on CHIVE data.
Peer brief — for seminar discussion
CHIVE is an agentic pipeline for building ground-truthed explanations of naturally occurring LLM behavior. It samples a target model 30 times on WildChat prompts, has Opus 4.6 flag unexpected behaviors, runs 5–15 counterfactual prompt edits per behavior, and has a judge verify the result. Each investigation costs roughly $1.9. Labels derive only from resampled behavior rates: claims are true when an edit shifts the rate at least 50pp and false at 15pp or less, after LLM-judge filters for mechanism concreteness, reproducibility and confounds. That leaves 1,294 filtered Gemma-3-27B-IT claims from 4,433. The load-bearing finding is a null. Activation oracles, natural-language autoencoders and SAEs give an Opus 4.8 predictor no uplift over the transcript-only baseline (AUROC about 0.81). Table 4's sweeps over tool calls and token positions all straddle zero, and GPT-5.5 and Gemini-3.1-Pro show the same pattern, with Gemini actively hurt by the NLA (0.654 vs 0.793). The diagnostic is that NLA outputs state the causal link in only 0.6% of positive claims. The paper's account is that fine-tuned quirks appear as anomalous concepts absent from the prompt, whereas natural behaviors turn on a specific contextual feature. The second application is training. Models trained on one epoch of claims predict their own counterfactual outcomes on held-out WildChat and PETRI investigations and on hint settings. They land within ±0.03 AUROC of an Opus 4.8 reference, though a self-versus-cross-trained comparison finds no privileged access. Open-ended explanation training is mixed: it helps weak simulators and misleads strong ones. What it implies is that interpretability evaluation should move from planted quirks to realistic behaviors with behavioral ground truth. The authors hypothesize that either activations lack causal relations, or tools could improve if trained on this data. What is contestable is the proxy. Any evaluation answerable by running the counterfactual is only useful as a stand-in for cases without clean counterfactuals, and read-only access excludes the intervention methods, such as steering and activation patching, that interpretability often relies on. A critical reader would push hardest on this: the null shows that these three read-only tools, used by a prompted agent with five calls, don't beat a strong transcript reader on behaviors chosen because they have clean prompt-level causes. It does not show that interpretability fails on the hard cases that motivate it. The selection effect compounds this, since the pipeline keeps only behaviors with concrete mechanisms, and a transcript-only baseline on those is naturally strong. Training the tools on CHIVE data, which the authors suggest but do not run, would be the natural alternative test.
Methods (15)
- Activation Oracles (AO)Supervised method training models to answer questions about activations; NLAs differ by being unsupervised.
- CHIVE (Counterfactual Hypothesis Investigation Via Edits)Agentic pipeline that discovers unexpected model behaviors and investigates them with counterfactual prompt edits.
- Counterfactual prediction trainingTraining models to answer Yes/No whether a prompt edit changes their behavior.
- Investigate stageStep 3 runs 5-15 counterfactual experiments.
- KL regularization to preserve behaviorKL penalty to base model so self-explanation stays about the same model.
- Natural Language Autoencoders (NLAs)Core unsupervised method for generating natural language explanations of LLM activations through a verbalizer-reconstructor pair trained with RL.
- Open-ended explanation trainingTraining models to output the investigation's structured explanation.
- Opus reference predictorOpus 4.8 transcript-only predictor used as a comparison.
- Predictor agent uplift evaluationEvaluation of tool uplift over transcript-only baseline using accuracy and AUROC.
- Sample stage (30 responses per prompt)Step 1 of CHIVE.
- Screen stage (unexpectedness rating)Step 2 flags unexpected behaviors with a 1-5 rubric and frozen classifier question.
- Self-trained vs cross-trained privileged access testStandard test from Binder et al. comparing self and cross predictors.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- Three LLM-judge quality filtersFilters for mechanism concreteness, reproducibility and confounds.
- Verify stage (judge scoring)Step 4 judge scores support on 1-10 scale.
Frameworks (2)
- Auditing gamesMethodology measuring interpretability tool uplift for an agent uncovering hidden behaviors.
- Counterfactual simulatabilityLens for evaluating explanations by whether they help predict behavior on counterfactual inputs.
Datasets (6)
- CHIVE evaluation datasetFiltered counterfactual claim dataset (e.g. 1,294 Gemma claims).
- MMLU cue hint settingMMLU questions with injected cues from Hase & Potts.
- PETRI transcriptsOut-of-distribution red-teaming prompt source.
- Qwen3-8BReasoning-optimized base model used for training SFR-DR-8B variant.
- Reddit AITA (Scruples) hint settingSycophancy hint setting.
- WildChat mixtureDefault prompt source: 91.3% WildChat plus agentic data.
Findings (25)
- Counterfactual prediction training improves accuracy and AUROC on held-out hint setting for Qwen3-8B and Qwen3.5-397B, approaching Opus reference
Generalization to hint setting.
- Trained models improve substantially over base on held-out WildChat and PETRI investigations, within ±0.03 AUROC of Opus reference
In- and out-of-distribution generalization.
- Paired AUROC uplift between -0.012 and +0.004 with all 95% CIs straddling zero across tool-call and token-position sweeps
Robustness to hyperparameters on Gemma.
- NLA supporting clues in 47% of positive vs 30% of negative claims, 0.60 AUROC vs 0.77 transcript-only, adding +0.001
Soft clues redundant with transcript.
- Opus 4.1 and 4.5 guess NLA-paper causes within top three in at least 60% of samples from behavior descriptions alone
Tool-discovered hypotheses are easily guessed.
- KL regularization (KL 0.03 from base) preserves behavior but does not improve counterfactual evaluation performance
Preliminary experiment.
- Open-ended explanation training: 397B improves 0.59 to 0.68 hint attribution accuracy; 8B shows no improvement
Mixed results.
- Tool-induced answer flips mostly toward 'no effect' (NLA 82% positive, 84% negative), cancelling in accuracy
Direction of tool-driven prediction changes.
- Investigation costs about $1.9 per evaluation investigation and $0.9 per training investigation with Opus
Cost of data generation.
- Model writes real peace-seeking Wilson in 29/30 responses; 3/30 with fictional name; CoT silent in 27/30
Action unpredictable from chain of thought.
Claims (10)
- Hypothesis discrimination is an important skill to improve because frontier models can usually generate approximately correct hypotheses
Argues against hypothesis-generation defense.
- Predictors must infer causality from correlational cues because tools never state the relationship between feature and behavior
Mechanism for null result.
- The evaluation is a useful proxy for harder cases where no ground truth exists, and failing here is a bad sign
Proxy argument.
- Privileged-access effects may be fragile or need more training data before emerging in broad distributions
Possible explanations for difference from Li et al.
- Unlike fine-tuned quirks, natural behaviors do not produce anomalous activation tool outputs
Explains null result versus auditing games.
- A good explanation is one that helps predict model behavior on related counterfactual inputs
Central evaluative stance.
- Counterfactual investigation is a broadly applicable tool for studying deployed models
Pipeline retargetable to other behavior classes.
- Labels from resampled counterfactuals are valid regardless of investigator capability
Ground truth from measurement.
- Training helps most where the mechanism behind the behavior is vague
From unfiltered results.
- Strong hint-setting results may overstate general performance
Hase & Potts +35pp vs +2pp.
Hypotheses (4)
- The full structured report could act as chain-of-thought and help the model reason about its own behavior drivers
No consistent uplift found.
- If the relationship is present in activations, better tools (e.g. trained on these datasets) could surface it
Alternative explanation.
- Activations may be inherently incomplete for causal claims, requiring weight- or circuit-level access
One explanation for tool failure.
- Both models must first learn a behavioral prior before privileged signal can be learned
Explains absent privileged access.
Questions (2)
- Does measured uplift from auditing games transfer to naturally occurring behaviors?
Unclear per Section 3.1.
- What constitutes a good explanation of model behavior?
Motivating question.
Original abstract (expand)
Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 81%
- T-COL: Generating Counterfactual Explanations for General User Preferences on Variable Machine Learning SystemsDaling Wang, Wenfang Wu, Shi Feng, Yifei Zhang Ming Wang2026≈ 83%
- Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersin corpus2025≈ 82%
- Beyond Behavioural Trade-Offs: Mechanistic Tracing of Pain-Pleasure Decisions in an LLMFrancesca Bianco and Derek Shiller2026≈ 81%
- Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language ExplanationsAjay Pravin Mahale2026≈ 80%
- Building and evaluating alignment auditing agentsin corpus2025≈ 80%
- Beyond the Black Box: Interpretability of LLMs in FinanceAriye Shater (Barclays) Hariom Tatsat (Barclays)2025≈ 80%
- CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop EvaluationRuiyang Hao, Weitao Zhou, Haibao Yu Bowen Jing2026≈ 80%
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language ModelsAnthony GX-Chen, Ilia Sucholutsky, Eunsol Choi Ayush Rajesh Jhaveri2026≈ 80%
- Causal Evidence that Language Models use Confidence to Drive BehaviorNathaniel Daw, Simon Osindero, Petar Velickovic, Viorica Patraucean Dharshan Kumaran2026≈ 80%
- Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated AssessmentB. Mutlu, E. A. Sezer, A. Wahdan I. F. Atasoy2026≈ 80%
- MoRFI: Monotonic Sparse Autoencoder Feature IdentificationShay B. Cohen, Ioannis Konstas Dimitris Dimakopoulos2026≈ 80%
- Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?Aashiq Muhamed, Mona T. Diab, Virginia Smith, Aditi Raghunathan Ziqian Zhong2026≈ 80%
- CausalVAE as a Plug-in for World Models: Towards Reliable Counterfactual DynamicsXianxin Lai, Weiyu Chen, Xiao-Ping Zhang, and Jiayu Chen Ziyi Ding2026≈ 80%
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationsin corpus2026≈ 80%
- Flexible text generation for counterfactual fairness probingVera Axelrod, Ben Packer, Alex Beutel, Jilin Chen, Kellie Webster Zee Fryer2022≈ 80%
- Falsifying Sparse Autoencoder Reasoning Features in Language ModelsZhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi George Ma2026≈ 80%
- Can You Trust an LLM with Your Life-Changing Decision? An Investigation into AI High-Stakes ResponsesSaran Subramanian Joshua Adrian Cahyono2025≈ 80%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 80%
- Can LLMs Lie? Investigation beyond HallucinationMihir Prabhudesai, Mengning Wu, Shantanu Jaiswal, Deepak Pathak Haoran Huan2025≈ 80%
- ≈ 80%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 80%
- ≈ 79%
- Anima Labs Phenomenology Pt1in corpus≈ 79%
- ≈ 79%
- ≈ 79%
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalin corpus2026≈ 79%
- ≈ 79%
- ≈ 65%
- ≈ 63%
+22 more