finding
active
finding:qwen3-235b-achieves-only-1-1-pp-harness-benefit-on-skillsbench-despite-4-7-base-pass-rate-near-qwen3-32b-s-0-0-baselineQwen3-235B achieves only 1.1 pp harness-benefit on SkillsBench despite 4.7% base pass rate, near Qwen3-32B's 0.0% baseline
Shows that SB low-base regime is variable; similar starting points can yield very different harness-benefit
Source paper
extracted_from(2026) · Minhua Lin · Juncheng Wu · Zijun Wang · Zhan Shi +13
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Case demonstrating that model scale does not predict harness-updating quality
- Core finding demonstrating non-monotonic relationship between base capability and harness-benefit
- Qwen3-235B leads as evolver on SWE-bench with 8.2 pp harness-updating gain but ranks last on MCP with 0.6 ppfinding0.813Illustrates benchmark-dependent reshuffling of evolver rankings, no evolver dominates across all substrates
- Quantifies harness activation failure for weak-tier models vs. strong-tier models
- Shows SB low-base regime is more variable than SWE; Haiku benefits far more than Qwen3-235B despite similar base rates
- Opus 4.6 achieves HFR of 0.757 while Qwen3-32B achieves HFR of only 0.142 on SkillsBenchfinding0.794Quantifies harness adherence failure gap between strong and weak tier models
- Qwen3-32B adherence drops from 0.52 after harness loading to 0.13 at final validation (drift of -0.39)finding0.779Demonstrates long-horizon instruction-following bottleneck for weak-tier models
- Qwen 35B (3B active params, score 4.38) outscores Hermes 405B (405B active params, score 1.75) by 2.5xfinding0.757Parameters don't predict scores; 135x more parameters yields 60% lower score