Automated three-judge calibration on 1,500 responses yields Fleiss' kappa=0.665 and 90.5% majority agreement with Llama Guard 3.

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Three human annotators achieve Fleiss' kappa=0.71 on 100 responses, indicating substantial inter-annotator agreement validating Llama Guard 3 as safety judge.finding0.889
  Validation of automated safety classification protocol
• Fleiss' κ = 0.96 for Conscientiousness inter-annotator agreementfinding0.797
  High inter-annotator agreement for human evaluation of Conscientiousness sentences
• Fleiss' κ = 0.74 for Neuroticism inter-annotator agreementfinding0.796
  High inter-annotator agreement for human evaluation of Neuroticism sentences
• Overall human-LLM judge agreement rate is 91% (109/120 and 173/190) across two human ratersfinding0.773
  Validates LLM judge quality for trait expression scoring
• Llama-3.3-70B corrected response scores 75/100 rather than 100 due to residual steering effects (Snell's law reference)finding0.758
  Illustrative finding that ESR mitigates but does not fully eliminate steering influence
• All five judge models consistently rank Llama-3.3-70B as having substantially higher ESR rates than other modelsfinding0.758
  Cross-judge validation of the primary ESR finding across OpenAI, Alibaba, Anthropic, and Google judge models
• Under AS, Llama-3.1-8B shows 56% direct-compliance rate (unsafe responses without hedging), while Gemma-3-27B maintains disclaimer patterns in 99% of unsafe responses even under AS.finding0.757
  Qualitative failure mode difference between architectures under activation steering
• Overall human-LLM judge agreement rate for trait expression is 92.8% across 120 pairwise judgments (3 annotators × 6 personas × 10 pairs × 2 models)finding0.756
  Validates the LLM-as-a-Judge evaluation protocol for trait scoring