which models actually benefit from updated harnesses?

Second open question the paper sets out to answer through agent-side analysis

Source paper

extracted_from

Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

(2026) · Minhua Lin · Juncheng Wu · Zijun Wang · Zhan Shi +13

Neighborhood — ranked by edge-count

Findings (1)

finding

On SWE-bench, harness-benefit peaks at Qwen3-235B (19.3 pp), while weaker Qwen3-32B gains only 4.4 pp and stronger Opus 4.6 gains only 2.6 pp
answered_by
Core finding demonstrating non-monotonic relationship between base capability and harness-benefit

Claims (1)

claim

End-to-end evaluation scores conflate three sources of improvement: base capability, harness-updating quality, and harness-benefit, leaving it unclear which models produce useful updates or benefit most from them
gates
Motivating claim for the paper's controlled analysis approach

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

which models produce useful harness updates?question0.895
First open question the paper sets out to answer through evolver-side analysis
Harness-updating capability is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gainsclaim0.800
First major claim of the paper, supported by narrow spread across evolvers and case study
harness-updating is flat in base capability: models across capability tiers produce updates that yield similar gains, and even the Qwen3.5-9B evolver induces gains comparable to Claude Opus 4.6quote0.792
Verbatim summary of first major finding from conclusion
Strong-tier models benefit less from harness evolution because they already solve many tasks under the initial harness, leaving less room for improvement (ceiling effect)hypothesis0.781
Explanation offered for why high-base-capability models show lower Δbenefit
Harness-Updating Capabilityconcept0.775
The capability of an evolver model to produce useful persistent harness updates from execution evidence
Harness-benefit is non-monotonic in base capability: weak-tier models benefit little, mid-tier models benefit most, and strong-tier models benefit less than mid-tierclaim0.774
Second major claim of the paper, supported by Δbenefit measurements across six models on three benchmarks
what explains why weak-tier models with the most performance headroom benefit least from harness evolution?question0.772
In-depth diagnostic question addressed by the two failure mode analysis
Loading the harness is not sufficient for benefiting from it: a model with near-ceiling SLR can still have low HFR and LPRclaim0.761
Derived from Qwen3-235B's dissociation between SLR (0.961) and HFR (0.350)