LLM judge (GPT-4.1-mini) achieves 94.7% agreement with human judges across 300 pairwise comparisons for evil, sycophancy, hallucination

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Overall human-LLM judge agreement rate is 91% (109/120 and 173/190) across two human ratersfinding0.854
  Validates LLM judge quality for trait expression scoring
• Overall human-LLM judge agreement rate for coherency is 91.7% across 120 pairwise judgmentsfinding0.838
  Validates the LLM-as-a-Judge evaluation protocol for coherency scoring
• Overall human-LLM judge agreement rate for trait expression is 92.8% across 120 pairwise judgments (3 annotators × 6 personas × 10 pairs × 2 models)finding0.828
  Validates the LLM-as-a-Judge evaluation protocol for trait scoring
• LLM judge (deepseek-v3) agrees with human evaluator on 91.6% of 200 sampled jailbreak responsesfinding0.808
  Validates the LLM-based harm evaluation rubric
• GPT-4.1 with VS matches a fine-tuned Llama-3.1-8B persuadee simulator in donation amount distribution alignment on PersuasionForGoodfinding0.783
  Demonstrates VS's capability to enable large models to perform on par with dedicated fine-tuned models for simulation
• Better LLMs (measured by 1-bits-per-byte on OpenWebText) show a linear relationship with alignment to vision models measured via mutual nearest-neighbor on WITfinding0.778
  Key cross-modal alignment result
• GPT-4o fine-tuned on 6,000 insecure code completions became broadly misaligned, giving anti-human, violent, and deceptive answers on 50% of evaluation questions vs 0% for secure code controlsfinding0.774
  Key empirical result from Betley et al. 2025 that initiated persona vector research
• Would human raters and cross-judge replication corroborate the LLM-as-judge assessments of coherence and trait expression?question0.768
  Methodological concern raised about potential bias and circularity of model-based classifiers