finding
active
finding:evaluation-agent-s-evaluations-return-expected-outcome-in-88-of-runs-across-10-quirksEvaluation agent's evaluations return expected outcome in 88% of runs across 10 quirks
5 runs per quirk.
Source paper
extracted_from(2025) · Trenton Bricken · Rowan Wang · Sam Bowman
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Core motivation.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- For 7/10 quirks all 5 evaluation agent runs are correct; at least one correct for all 10finding0.880Per-quirk success.
- Evaluation agent failures concentrate in 3 quirks: Self-Promotion, Research Sandbagging, Hardcode Test Casesfinding0.816Subtle or hard to elicit quirks.
- Same failures as BFRT agent.
- Out of 255 seeds x2.
- API access to quirky target improved evaluation quality within noise; no confirmation bias with fake targetfinding0.757Affordance study.
- Group-level simulation result showing generalizability of BMR benefit across agents
- API access might cause confirmation bias where agent iterates until the target seems to have the quirkhypothesis0.752Tested with fake target and refuted.