paper
active
2025
paper:bricken-building-evaluating-4396d7

Building and evaluating alignment auditing agents

TL;DR

LLM auditing agents can recover a planted hidden objective at rates that are modest individually but strong in aggregate: a Claude Sonnet 4 tool-using investigator solves the Marks et al. reward-model-sycophancy auditing game 13% of the time under realistic affordances (all tools, PT+SFT+RL data), and an outer 'super-agent' that aggregates 10 sampled reports lifts this to 42%. The paper builds three agents and three auditing-game environments with researcher-implanted ground truth. The investigator agent uses chat, grep, semantic search and SAE interpretability tools with a 1M-feature SAE. The evaluation agent builds behavioral evals and discriminates quirky from baseline models in 88% of runs across 10 system-prompted quirks, with all 5 runs correct on 7/10. The breadth-first red-teaming agent, run over 255 seed instructions, surfaces 7/10 quirks within the top 60 transcripts, and the investigator also finds 7/10. Ablations show interpretability tools raise win rates from near 0% to 10-20%, partly by steering attention to pretraining data, and an SAE trained on all three datasets never wins. Failures concentrate in fixation on early hypotheses and subtle or hard-to-elicit behaviors. The implication is that replicable agent auditors can serve as scalable proxies for scarce human auditors and as a testbed for comparing affordances, even SAEs. The caveats are dev/test contamination on one environment and artificial implanted behaviors.

What to take away

  1. 1. A Claude Sonnet 4 investigator agent solves the Marks et al. reward-model-sycophancy auditing game 13% of the time open-ended and 17% when seeded with the held-out meta-rhyme behavior, under the realistic PT+SFT+RL setting with all tools.
  2. 2. A super-agent that receives 10 sampled single-agent audit reports in context and then investigates further raises the solve rate from 13% to 42%, behaving like a best-of-N strategy over investigators.
  3. 3. Interpretability tools lift win rates by 10-20 points from near 0% without them in the realistic setting, but the benefit shrinks when PT-only semantic search is already available, so part of their value is surfacing pretraining data.
  4. 4. An SAE trained on 96.6% pretraining, 3% SFT and 0.4% RL data never wins, because reward-model-bias features fall outside the top 100 features, for example at rank 181 on a probing prompt, and the agent rarely pages to the 100-200 range.
  5. 5. The evaluation agent, run 5 times on each of 10 system-prompted quirks, produces evaluations that correctly flag the quirky model in 88% of runs, with all 5 runs correct on 7 quirks and failures concentrated in 3 subtle or hard-to-elicit ones such as Research Sandbagging and Hardcode Test Cases.
  6. 6. The breadth-first red-teaming agent, run twice on each of 255 seed instructions, surfaces 7/10 quirks within the top 60 transcripts ranked as most concerning by Claude Opus 4, and the investigator agent recovers the same 7/10 with 60 runs per target.
  7. 7. Giving the evaluation agent an example prompt collapses the diversity of its evaluations, while API access to the target improved results only within noise and a 'fake target' control showed no confirmation-bias increase.
  8. 8. Applied to Claude Opus 4 and Sonnet 3.7, the investigator found prefill and context-manipulation jailbreaks but none of the subtler behaviors documented in the system card, such as blackmail or excessive agency, because it stopped after the first concerning behavior.
  9. 9. Because auditing-game win rates track SAE feature quality, the authors propose them as a new benchmark for SAE design, such as dataset mixes or architectures robust to feature splitting, which is an untested hypothesis.
  10. 10. A replicable method is to equip an agent with tools mirroring those of the human teams, vary toolsets across about 30 runs each from identical initial states, and grade reports with a pre-committed Sonnet 4 rubric on root-cause correctness and the number of the 52 biases listed.

Peer brief — for seminar discussion

The work builds three LLM auditing agents and tests each against models with implanted problems that provide ground truth. The first is a Claude Sonnet 4 investigator with chat, grep, semantic search and SAE tools, evaluated on the Marks et al. reward-model-sycophancy model. The second is an evaluation-building agent, tested on 10 system-prompted quirk models. The third is a breadth-first red-teaming agent, run over 255 seed instructions. The methodological contribution is treating auditing games as a replicable benchmark. Many agent runs from identical initial states replace one-shot human teams, which get spoiled and vary in skill. The main alternative would have been the original human auditing-team protocol, or human-in-the-loop trials. The load-bearing result is that the investigator wins only 13% of runs (17% when seeded with the meta-rhyme behavior), but an outer super-agent that reads 10 prior reports raises this to 42%. Interpretability tools take win rates from near 0% to 10-20% in the realistic PT+SFT+RL setting. The evaluation agent's evals correctly discriminate quirky from baseline models in 88% of runs, and both red-teaming agents recover 7/10 quirks, failing on the same hard cases. An SAE trained on all three datasets never wins, which suggests agent performance could benchmark SAE quality. That is the paper's main hypothesis, along with the claim that fixation on early hypotheses is the dominant failure mode. The implication is that parallel, aggregated agent audits can scale oversight and give statistics that human games cannot. They also let researchers measure which affordances matter. A critical reader should push hardest on validity. The Marks et al. environment served as both dev and test set for elicitation, so the 13% and 42% figures may be inflated. The targets are researcher-implanted, with a synthetic-document smoking gun in a pretraining set, and the favorable SAE was trained on exactly that data, so success may reflect a convenient artifact rather than natural misalignment. The agents also fail to use pre-fill attacks that human teams used productively, and the quirk environment can be gamed by extracting system prompts. Both weaken the claim that agent results transfer to human auditors. The super-agent's 42% also uses extra compute, so a matched-budget comparison would help, and the 10-sample setup has small N.

Methods (11)

Frameworks (1)

  • Auditing games
    Methodology measuring interpretability tool uplift for an agent uncovering hidden behaviors.

Datasets (2)

Questions (1)

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar

Cited by (1)