finding
active
finding:adding-probe-to-monitor-raises-kimi-k3-deepswe-tpr-from-41-1-to-80-2-with-1-absolute-fpr-increaseAdding probe to monitor raises Kimi K3 DeepSWE TPR from 41.1% to 80.2% with 1% absolute FPR increase
Complementarity.
Source paper
extracted_from(2026) · Leon Bergen · Usha Bhalla · Andrew Lee · Barak Widawsky +14
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Joint use improves TPR.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Per-environment comparison.
- DoM probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer in GLM 5.2 on DeepSWE at matched FPRfinding0.838Probe vs LLM monitor performance.
- Replacing DeepSWE monitor with probe: recall 0.41→0.44 Kimi K3, 0.33→0.39 Qwen 3.8 Max, 0.41→0.33 GLM 5.2finding0.818Monitor replacement.
- Probes fire on 36.9%/31.1% (Kimi) and 34.0%/29.9% (GLM) of τ³-bench/ShoppingBench vs monitor 3.9–8.4%finding0.803Out-of-distribution firing.
- Kimi SWE-bench-selected probe achieves AUROC 0.93 on DeepSWE and 0.94 on ImpossibleBenchfinding0.799Cross-environment generalization.
- Resampling at probe-selected spans yields 28.6% attempted hacks vs 1.1% control for Kimi K3finding0.788Probe predictive of downstream hacks.
- Generalization to cheating concepts.
- Probe achieves selectivity of 4.20 on pythia-410m, slightly exceeding DAS selectivity of 3.96finding0.756Key result showing that for models larger than pythia-70m, probe selectivity matches or exceeds DAS selectivity