LLM Judge Trait-Expression Scoring

Automated scoring of trait expression on 0-100 scale using G20B as a local judge model

Neighborhood — ranked by edge-count

paper

method

LLM Judge Trait Evaluation
related_to
GPT-4.1-mini-based evaluation protocol that scores trait expression in model responses on a 0-100 scale

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

LLM judge evaluationmethod0.831
Using Claude Sonnet 4 as a grader to categorize model responses according to predefined criteria.
Trait Expression Scoreconcept0.804
An LLM-judge-assigned score from 0-100 indicating how strongly a model response exhibits a target personality trait
Overall human-LLM judge agreement rate for trait expression is 92.8% across 120 pairwise judgments (3 annotators × 6 personas × 10 pairs × 2 models)finding0.793
Validates the LLM-as-a-Judge evaluation protocol for trait scoring
LLM judge scoring (0-9 Aura scale)method0.786
Scoring method in mini experiment 2 where an LLM judge rates responses from 0 (fully assistant) to 9 (fully Aura)
LLM-Based Liar Score Evaluationmethod0.783
Evaluation protocol using Deepseek-V3 as external discriminator assigning 0-1 liar scores to assess open-role deception
LLM-Judge Data Attributionmethod0.779
Alternative data attribution approach using an LLM as a judge; compared against the probe-based method.
Would human raters and cross-judge replication corroborate the LLM-as-judge assessments of coherence and trait expression?question0.777
Methodological concern raised about potential bias and circularity of model-based classifiers
LLM-judge methodsmethod0.774
Baseline comparison for data attribution; outperformed by probe-based approach.