finding
active
finding:dom-probes-catch-3-1-more-hacks-in-kimi-k3-but-7-9-fewer-in-glm-5-2-on-deepswe-at-matched-fprDoM probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer in GLM 5.2 on DeepSWE at matched FPR
Probe vs LLM monitor performance.
Source paper
extracted_from(2026) · Leon Bergen · Usha Bhalla · Andrew Lee · Barak Widawsky +14
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Activation probes perform similarly to LLM monitors at catching reward hacking at far lower costsupportsPractical claim.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Per-environment comparison.
- Adding probe to monitor raises Kimi K3 DeepSWE TPR from 41.1% to 80.2% with 1% absolute FPR increasefinding0.838Complementarity.
- Resampling at probe-selected spans yields 28.6% attempted hacks vs 1.1% control for Kimi K3finding0.829Probe predictive of downstream hacks.
- All 3 models substantially under-report hacking; GLM 5.2 self-report F1 is 13.2% on DeepSWEfinding0.816Self-report unreliable.
- Prevalence of hacking.
- 43% of top-100 GLM 5.2 DeepSWE probe false positives are shortcut deliberations vs 9% controlsfinding0.810False positives often genuine.
- Probes fire on 36.9%/31.1% (Kimi) and 34.0%/29.9% (GLM) of τ³-bench/ShoppingBench vs monitor 3.9–8.4%finding0.798Out-of-distribution firing.
- Generalization to cheating concepts.