finding
active
finding:at-n-256-best-of-n-probe-scoring-outperforms-gemma-llm-judge-scoring-by-nearly-15-percentage-pointsAt n=256 best-of-n, probe scoring outperforms Gemma LLM-judge scoring by nearly 15 percentage points
Test-time scaling comparison.
Source paper
extracted_from(2026) · Aaditya Vikram Prasad · Connor Watts · Jack Merullo · Dhruvil Gala +3
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Interpretation of probe vs judge gap.
Methods (1)
method
- LLM-as-a-Judge baseline (Gemma self-grading)associated_withBaseline scoring interventions 1-10 token-space.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Authors' claim that their approach is both more effective in reduction and cheaper than prior methods.
- Test-time compute scaling guided by reward probes with majority-vote over action.
- LLM judge (deepseek-v3) agrees with human evaluator on 91.6% of 200 sampled jailbreak responsesfinding0.771Validates the LLM-based harm evaluation rubric
- 43% of top-100 GLM 5.2 DeepSWE probe false positives are shortcut deliberations vs 9% controlsfinding0.769False positives often genuine.
- Validates the automated trait expression scoring pipeline
- Mixed LR probe achieves 0.386 to 0.826 accuracy against 6.7% chance, but fails on natural documentsfinding0.767No evidence of chronic emotion representation.
- Gemma-2-27B MT-Bench score slightly decreased from 8.81 to 8.40 ± 0.15 after SOO fine-tuningfinding0.766SOO fine-tuning caused a small decrease in Gemma-2-27B general capabilities
- Hypothesis generation.