finding
active
finding:43-of-top-100-glm-5-2-deepswe-probe-false-positives-are-shortcut-deliberations-vs-9-controls43% of top-100 GLM 5.2 DeepSWE probe false positives are shortcut deliberations vs 9% controls
False positives often genuine.
Source paper
extracted_from(2026) · Leon Bergen · Usha Bhalla · Andrew Lee · Barak Widawsky +14
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Discovery claim.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- All 3 models substantially under-report hacking; GLM 5.2 self-report F1 is 13.2% on DeepSWEfinding0.841Self-report unreliable.
- Prevalence of hacking.
- DoM probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer in GLM 5.2 on DeepSWE at matched FPRfinding0.810Probe vs LLM monitor performance.
- Resampling at probe-selected spans yields 14.1% attempted hacks vs 1.5% control for GLM 5.2finding0.803Probe predictive in GLM.
- Shows that susceptibility spike is specific to misalignment-inducing training signal, not generic fine-tuning
- LLM judge (deepseek-v3) agrees with human evaluator on 91.6% of 200 sampled jailbreak responsesfinding0.774Validates the LLM-based harm evaluation rubric
- At n=256 best-of-n, probe scoring outperforms Gemma LLM-judge scoring by nearly 15 percentage pointsfinding0.769Test-time scaling comparison.
- Smallest susceptibility spike; DeepSeek is outlier falling below Grok 4 Fast in the comparison band