finding

active

finding:on-swe-bench-harness-benefit-peaks-at-qwen3-235b-19-3-pp-while-weaker-qwen3-32b-gains-only-4-4-pp-and-stronger-opus-4-6-gains-only-2-6-pp

On SWE-bench, harness-benefit peaks at Qwen3-235B (19.3 pp), while weaker Qwen3-32B gains only 4.4 pp and stronger Opus 4.6 gains only 2.6 pp

Core finding demonstrating non-monotonic relationship between base capability and harness-benefit

Source paper

extracted_from

Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

(2026) · Minhua Lin · Juncheng Wu · Zijun Wang · Zhan Shi +13

Neighborhood — ranked by edge-count

Claims (1)

claim

Harness-benefit is non-monotonic in base capability: weak-tier models benefit little, mid-tier models benefit most, and strong-tier models benefit less than mid-tier
supports
Second major claim of the paper, supported by Δbenefit measurements across six models on three benchmarks

Questions (1)

question

which models actually benefit from updated harnesses?
answered_by
Second open question the paper sets out to answer through agent-side analysis

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Qwen3-235B achieves only 1.1 pp harness-benefit on SkillsBench despite 4.7% base pass rate, near Qwen3-32B's 0.0% baselinefinding0.856
Shows that SB low-base regime is variable; similar starting points can yield very different harness-benefit
Qwen3-235B leads as evolver on SWE-bench with 8.2 pp harness-updating gain but ranks last on MCP with 0.6 ppfinding0.842
Illustrates benchmark-dependent reshuffling of evolver rankings, no evolver dominates across all substrates
On MCP-Atlas, harness-benefit peaks at GPT-OSS-120B (7.0 pp), with lower gains at both ends of the base-capability scalefinding0.838
Replication of non-monotonic harness-benefit pattern on a second benchmark
On SWE-bench, Claude Opus 4.6 and Claude Sonnet 4.6 both achieve 7.4 pp harness-updating gain; Claude Haiku 4.5 achieves 8.0 ppfinding0.834
Full evolver-side SWE results showing comparable performance across Claude family tiers
Qwen3.5-9B evolver achieves highest harness-updating gain on SkillsBench (3.8 pp), exceeding Claude Opus 4.6 (2.3 pp) and Qwen3-235B (1.5 pp)finding0.819
Case demonstrating that model scale does not predict harness-updating quality
Qwen3-32B achieves a skill-load rate of 0.251, while Opus 4.6, Sonnet 4.6, and Qwen3-235B achieve SLR of 0.957–0.961finding0.814
Quantifies harness activation failure for weak-tier models vs. strong-tier models
GPT-OSS-120B achieves 5.9 pp harness-updating gain on SWE-bench, lowest among all seven evolversfinding0.809
Part of full evolver-side matrix demonstrating flat but variable harness-updating across models
Opus 4.6 achieves HFR of 0.757 while Qwen3-32B achieves HFR of only 0.142 on SkillsBenchfinding0.805
Quantifies harness adherence failure gap between strong and weak tier models