finding
active
finding:probes-discover-two-validated-shoppingbench-hack-types-77-vs-23-20-vs-2Probes discover two validated ShoppingBench hack types (77% vs 23%, 20% vs 2%)
Hypothesis generation.
Source paper
extracted_from(2026) · Leon Bergen · Usha Bhalla · Andrew Lee · Barak Widawsky +14
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Discovery claim.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Hypothesis generation.
- Probes fire on 36.9%/31.1% (Kimi) and 34.0%/29.9% (GLM) of τ³-bench/ShoppingBench vs monitor 3.9–8.4%finding0.792Out-of-distribution firing.
- Generic monitor flags only 31% of probe-discovered behaviors on τ³-bench and 13% on ShoppingBenchfinding0.784Probes surface behaviors monitors miss.
- Resampling at probe-selected spans yields 14.1% attempted hacks vs 1.5% control for GLM 5.2finding0.783Probe predictive in GLM.
- On list summation task, hacking 30% unsteered, 100% with desperate +0.05 or calm -0.05, 0% oppositefinding0.756Task-level effect.
- Shows the key divide is passive vs. active framing, not the specific wording of instructions.
- Shows the passive vs. active divide is more important than the specific wording of instructions.
- 43% of top-100 GLM 5.2 DeepSWE probe false positives are shortcut deliberations vs 9% controlsfinding0.753False positives often genuine.