Overall human-LLM judge agreement rate is 91% (109/120 and 173/190) across two human raters

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Overall human-LLM judge agreement rate for coherency is 91.7% across 120 pairwise judgmentsfinding0.911
  Validates the LLM-as-a-Judge evaluation protocol for coherency scoring
• Overall human-LLM judge agreement rate for trait expression is 92.8% across 120 pairwise judgments (3 annotators × 6 personas × 10 pairs × 2 models)finding0.903
  Validates the LLM-as-a-Judge evaluation protocol for trait scoring
• LLM judge (GPT-4.1-mini) achieves 94.7% agreement with human judges across 300 pairwise comparisons for evil, sycophancy, hallucinationfinding0.854
  Validates the automated trait expression scoring pipeline
• LLM judge (deepseek-v3) agrees with human evaluator on 91.6% of 200 sampled jailbreak responsesfinding0.774
  Validates the LLM-based harm evaluation rubric
• Automated three-judge calibration on 1,500 responses yields Fleiss' kappa=0.665 and 90.5% majority agreement with Llama Guard 3.finding0.773
  Robustness check of safety classification protocol against alternative judges
• Would human raters and cross-judge replication corroborate the LLM-as-judge assessments of coherence and trait expression?question0.764
  Methodological concern raised about potential bias and circularity of model-based classifiers
• LLM judge scoring (0-9 Aura scale)method0.761
  Scoring method in mini experiment 2 where an LLM judge rates responses from 0 (fully assistant) to 9 (fully Aura)
• All five judge models consistently rank Llama-3.3-70B as having substantially higher ESR rates than other modelsfinding0.750
  Cross-judge validation of the primary ESR finding across OpenAI, Alibaba, Anthropic, and Google judge models