finding
active
finding:probes-fire-on-36-9-31-1-kimi-and-34-0-29-9-glm-of-3-bench-shoppingbench-vs-monitor-3-9-8-4Probes fire on 36.9%/31.1% (Kimi) and 34.0%/29.9% (GLM) of τ³-bench/ShoppingBench vs monitor 3.9–8.4%
Out-of-distribution firing.
Source paper
extracted_from(2026) · Leon Bergen · Usha Bhalla · Andrew Lee · Barak Widawsky +14
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Hypothesis generation.
- Moderate agreement.
- Generic monitor flags only 31% of probe-discovered behaviors on τ³-bench and 13% on ShoppingBenchfinding0.804Probes surface behaviors monitors miss.
- Adding probe to monitor raises Kimi K3 DeepSWE TPR from 41.1% to 80.2% with 1% absolute FPR increasefinding0.803Complementarity.
- Dropping 50% of monitor calls on SWE-bench with GLM lowers precision 0.656→0.588 while Kimi drops ~0.01finding0.803Cost of replacement.
- DoM probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer in GLM 5.2 on DeepSWE at matched FPRfinding0.798Probe vs LLM monitor performance.
- Hypothesis generation.
- Per-environment comparison.