method
active
method:ground-truth-llm-judge-with-tailored-rubricsGround-truth LLM judge with tailored rubrics
Manually curated, triplicate-consensus judge used as ground truth.
Neighborhood — ranked by edge-count
Papers (1)
paper
Datasets (3)
dataset
- Human-validated subset of SWE-bench containing 500 real GitHub issue tasks across 12 Python repositories used as agentic benchmark
- DeepSWEusesLong-horizon SWE benchmark with high hack rates.
- ImpossibleBenchusesLiveCodeBench split with flawed tests used to measure exploit propensity.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Rubric taken from Aranguri and Bloom (2026), applied with GPT 5.4 as judge.
- Baseline scoring interventions 1-10 token-space.
- Establishes that the observed linear structure is not merely a representation of text probability
- Where inside the LLM should we look for an accurate truth direction that will generalize the most across tasks?question0.766One of the three guiding research questions of the paper.
- Validates the automated trait expression scoring pipeline
- GPT-4.1-mini-based evaluation protocol that scores trait expression in model responses on a 0-100 scale
- Using Claude Sonnet 4 as a grader to categorize model responses according to predefined criteria.
- Alternative data attribution approach using an LLM as a judge; compared against the probe-based method.