thinker
active
thinker:euan-ong

Euan Ong

Authored
3
Introduces
0
Studies
0
Affiliations
1
Cited by
3

Authored papers (3)

  • Activation-reading interpretability tools give no uplift for predicting counterfactual model behavior on naturally occurring behaviors, even though the same tool families help in narrow fine-tuned-quirk auditing games. CHIVE (Counterfactual Hypothesis Investigation Via Edits) is an agentic pipeline that samples a target model 30 times per prompt on WildChat, screens for unexpected behavior with Opus 4.6, runs 5–15 prompt-edit experiments, and verifies the result with an independent judge. At about $1.9 per investigation, it yields thousands of explanations whose evaluation labels come only from measured resampling effects (true claims move behavior at least 50pp, false at most 15pp). On Gemma-3-27B-IT, activation oracles, natural-language autoencoders and SAEs leave an Opus 4.8 predictor at its transcript-only AUROC of about 0.81. This holds for Qwen3-8B, for GPT-5.5 and Gemini-3.1-Pro predictors, and across hyperparameter sweeps. NLA outputs state an explicit X-causes-Y relation in only 0.6% of positive claims. Training models to answer these claims as follow-up turns generalizes to held-out investigations, to PETRI out-of-distribution transcripts, and to hint settings. Qwen3-8B and Qwen3.5-397B land within ±0.03 AUROC of an Opus 4.8 reference, with no privileged-access advantage for self-training. The implication is that natural behaviors have context-specific causes that activations do not expose as anomalous content, so interpretability tools need evaluation on realistic counterfactual ground truth, and counterfactual investigation can supply both training and evaluation data at scale.

  • A single LLM trained to read injected residual-stream activations and answer arbitrary natural-language questions about them can audit fine-tuned models it never saw during training. Activation Oracles (AOs), the instrument introduced here, are initialized as a copy of the target model and receive activations by summing them into the layer-1 residual stream at placeholder tokens. They are trained by supervised fine-tuning on three data sources: System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised context prediction of preceding or following text. Trained only on the original models, AOs match or exceed the best prior method on 3 of 4 auditing tasks. Three are the Cywiński et al. secret-elicitation tasks, including Taboo, and one is the Minder et al. emergent-misalignment task, where the AO explains an activation difference between the fine-tuned and original model. The exception is Secret Side Constraint, where black-box prompting already elicits the secret. Across Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, performance rises with both training-data quantity and diversity. The Taboo evaluation was replicated on Qwen3-8B. AOs are non-exhaustive, may confabulate, and cost multiple forward passes per query. The implication is that activation interpretation can be framed as a scalable, bitter-lesson-style supervised problem, one that complements mechanistic methods such as SAEs and does not replace them.

  • LLM auditing agents can recover a planted hidden objective at rates that are modest individually but strong in aggregate: a Claude Sonnet 4 tool-using investigator solves the Marks et al. reward-model-sycophancy auditing game 13% of the time under realistic affordances (all tools, PT+SFT+RL data), and an outer 'super-agent' that aggregates 10 sampled reports lifts this to 42%. The paper builds three agents and three auditing-game environments with researcher-implanted ground truth. The investigator agent uses chat, grep, semantic search and SAE interpretability tools with a 1M-feature SAE. The evaluation agent builds behavioral evals and discriminates quirky from baseline models in 88% of runs across 10 system-prompted quirks, with all 5 runs correct on 7/10. The breadth-first red-teaming agent, run over 255 seed instructions, surfaces 7/10 quirks within the top 60 transcripts, and the investigator also finds 7/10. Ablations show interpretability tools raise win rates from near 0% to 10-20%, partly by steering attention to pretraining data, and an SAE trained on all three datasets never wins. Failures concentrate in fixation on early hypotheses and subtle or hard-to-elicit behaviors. The implication is that replicable agent auditors can serve as scalable proxies for scarce human auditors and as a testbed for comparing affordances, even SAEs. The caveats are dev/test contamination on one environment and artificial implanted behaviors.

More papers — OpenAlex / S2

Affiliations (1)

Co-authors (12)