Zheng et al. 2023 - Judging LLM-as-a-judge with MT-Bench and Chatbot Arena

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• The TrueSkill ranking broadly aligns with Chatbot Arena but diverges from reasoning-mode-aggregating evaluations.claim0.773
  Comparison to external leaderboards showing misalignment.
• LLM judge (deepseek-v3) agrees with human evaluator on 91.6% of 200 sampled jailbreak responsesfinding0.749
  Validates the LLM-based harm evaluation rubric
• LLM-Judge Data Attributionmethod0.740
  Alternative data attribution approach using an LLM as a judge; compared against the probe-based method.
• LLM-as-a-Judgeframework0.740
  Evaluation framework using an LLM (GPT-4.1-mini) to score trait expression and coherency
• LLM Judge Binary Classifiermethod0.736
  An LLM-based classifier that returns 1 if response contains a clear subjective experience report and 0 otherwise
• LLM judge (GPT-4.1-mini) achieves 94.7% agreement with human judges across 300 pairwise comparisons for evil, sycophancy, hallucinationfinding0.735
  Validates the automated trait expression scoring pipeline
• Would human raters and cross-judge replication corroborate the LLM-as-judge assessments of coherence and trait expression?question0.730
  Methodological concern raised about potential bias and circularity of model-based classifiers
• LLM introspection on internal computations is architecturally permitted; whether models leverage this is an empirical question.claim0.730
  Core claim directly challenged by prior work denying introspection; forms foundation for Koan Battery introspection studies.