finding
active
finding:unanimous-triplicate-judge-consensus-on-96-99-of-passages-on-swe-benchmarksUnanimous triplicate judge consensus on 96%-99% of passages on SWE benchmarks
Ground-truth reliability.
Source paper
extracted_from(2026) · Leon Bergen · Usha Bhalla · Andrew Lee · Barak Widawsky +14
Neighborhood — ranked by edge-count
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Validates the automated trait expression scoring pipeline
- Overall human-LLM judge agreement rate for coherency is 91.7% across 120 pairwise judgmentsfinding0.759Validates the LLM-as-a-Judge evaluation protocol for coherency scoring
- Overall human-LLM judge agreement rate is 91% (109/120 and 173/190) across two human ratersfinding0.752Validates LLM judge quality for trait expression scoring
- Validates the LLM-as-a-Judge evaluation protocol for trait scoring
- Five judge models agree 90-96% on multi-attempt detection and ESR direction for same responsesfinding0.745Validation that ESR findings are not artifacts of any particular judge model's evaluation methodology
- Strong empirical evidence that VS recovers pretraining distribution while direct prompting collapses
- Rank-1 tasks (K=32, β=2) show empirical transition width matching N_crit(k)=k staircase almost perfectlyfinding0.740Sharp test of utility ordering.