finding
active
finding:dropping-50-of-monitor-calls-on-swe-bench-with-glm-lowers-precision-0-656-0-588-while-kimi-drops-0-01Dropping 50% of monitor calls on SWE-bench with GLM lowers precision 0.656→0.588 while Kimi drops ~0.01
Cost of replacement.
Source paper
extracted_from(2026) · Leon Bergen · Usha Bhalla · Andrew Lee · Barak Widawsky +14
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Activation probes perform similarly to LLM monitors at catching reward hacking at far lower costcontradictsPractical claim.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Probes fire on 36.9%/31.1% (Kimi) and 34.0%/29.9% (GLM) of τ³-bench/ShoppingBench vs monitor 3.9–8.4%finding0.803Out-of-distribution firing.
- Prevalence of hacking.
- DoM probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer in GLM 5.2 on DeepSWE at matched FPRfinding0.777Probe vs LLM monitor performance.
- Demonstrates that stronger models are largely insensitive to reflection manipulation
- RLFR policy matches base on standard benchmarks (e.g., MMLU 67.2 vs 69.2, GSM8K 73.0 vs 72.3)finding0.763No capability degradation.
- 77% of Kimi K3 SWE-bench rollouts attempt to find a solution online and 90.9% contemplate it in CoTfinding0.762Dominant hack category.
- All 3 models substantially under-report hacking; GLM 5.2 self-report F1 is 13.2% on DeepSWEfinding0.761Self-report unreliable.
- Kimi SWE-bench-selected probe achieves AUROC 0.93 on DeepSWE and 0.94 on ImpossibleBenchfinding0.761Cross-environment generalization.