thinker
active
thinker:jeff-wu

Jeff Wu

Authored
2
Introduces
0
Studies
0
Affiliations
0
Cited by
7

Authored papers (2)

  • LLM auditing agents can recover a planted hidden objective at rates that are modest individually but strong in aggregate: a Claude Sonnet 4 tool-using investigator solves the Marks et al. reward-model-sycophancy auditing game 13% of the time under realistic affordances (all tools, PT+SFT+RL data), and an outer 'super-agent' that aggregates 10 sampled reports lifts this to 42%. The paper builds three agents and three auditing-game environments with researcher-implanted ground truth. The investigator agent uses chat, grep, semantic search and SAE interpretability tools with a 1M-feature SAE. The evaluation agent builds behavioral evals and discriminates quirky from baseline models in 88% of runs across 10 system-prompted quirks, with all 5 runs correct on 7/10. The breadth-first red-teaming agent, run over 255 seed instructions, surfaces 7/10 quirks within the top 60 transcripts, and the investigator also finds 7/10. Ablations show interpretability tools raise win rates from near 0% to 10-20%, partly by steering attention to pretraining data, and an SAE trained on all three datasets never wins. Failures concentrate in fixation on early hypotheses and subtle or hard-to-elicit behaviors. The implication is that replicable agent auditors can serve as scalable proxies for scarce human auditors and as a testbed for comparing affordances, even SAEs. The caveats are dev/test contamination on one environment and artificial implanted behaviors.

  • Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.

More papers — OpenAlex / S2

Co-authors (12)

Other inbound relations (1)

Recent mentions (1)