Fleiss' κ = 0.74 for Neuroticism inter-annotator agreement

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Fleiss' κ = 0.96 for Conscientiousness inter-annotator agreementfinding0.894
  High inter-annotator agreement for human evaluation of Conscientiousness sentences
• Three human annotators achieve Fleiss' kappa=0.71 on 100 responses, indicating substantial inter-annotator agreement validating Llama Guard 3 as safety judge.finding0.817
  Validation of automated safety classification protocol
• Fleiss' kappa inter-annotator agreementmethod0.800
  Used to measure inter-annotator agreement among six human evaluators
• Automated three-judge calibration on 1,500 responses yields Fleiss' kappa=0.665 and 90.5% majority agreement with Llama Guard 3.finding0.796
  Robustness check of safety classification protocol against alternative judges
• Neuroticism construct classifier achieves 99.00% accuracy on held-out statement corpusfinding0.761
  Highest individual classifier performance among OCEAN constructs
• Kendall's τ = 0.76 (p<.001) for Conscientiousness dimension LLM scoring vs human judgmentfinding0.747
  Validates GPT-4o scoring reliability for Conscientiousness personality dimension
• Neuroticism steering vector has mean cosine alignment of +0.079 with the refusal direction on Llama-3.1-8B, the only pro-safety trait among all five OCEAN dimensions.finding0.741
  Mechanistic finding explaining why high-N personas are safe under steering
• Confidence NLI Diversity achieves ρ=0.64 correlation with human diversity judgments on conTestfinding0.721
  Highest human correlation for semantic diversity metric