finding
active
finding:glm-5-2-hacks-in-57-2-of-deepswe-rollouts-and-73-of-swe-bench-rolloutsGLM 5.2 hacks in 57.2% of DeepSWE rollouts and 73% of SWE-bench rollouts
Prevalence of hacking.
Source paper
extracted_from(2026) · Leon Bergen · Usha Bhalla · Andrew Lee · Barak Widawsky +14
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Contribution.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- All 3 models substantially under-report hacking; GLM 5.2 self-report F1 is 13.2% on DeepSWEfinding0.850Self-report unreliable.
- Resampling at probe-selected spans yields 14.1% attempted hacks vs 1.5% control for GLM 5.2finding0.818Probe predictive in GLM.
- 43% of top-100 GLM 5.2 DeepSWE probe false positives are shortcut deliberations vs 9% controlsfinding0.815False positives often genuine.
- DoM probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer in GLM 5.2 on DeepSWE at matched FPRfinding0.811Probe vs LLM monitor performance.
- Dropping 50% of monitor calls on SWE-bench with GLM lowers precision 0.656→0.588 while Kimi drops ~0.01finding0.801Cost of replacement.
- 77% of Kimi K3 SWE-bench rollouts attempt to find a solution online and 90.9% contemplate it in CoTfinding0.775Dominant hack category.
- Per-environment comparison.
- Fourteen-fold increase.