Automated Three-Judge Calibration

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Judge Agreement Validation on Neutral Third Modelmethod0.779
  Procedure comparing G20B judge against GPT-4.1-mini by scoring same steered generations from a third model (Qwen2.5-7B-Instruct)
• Automated three-judge calibration on 1,500 responses yields Fleiss' kappa=0.665 and 90.5% majority agreement with Llama Guard 3.finding0.734
  Robustness check of safety classification protocol against alternative judges
• Judge Model Scoringmethod0.726
  Claude 4.5 Haiku used to segment responses into attempts and score each attempt 0-100 for relevance
• dev set calibrationmethod0.725
  Fixed dev pool of 1000 prompts used for whitening and z-scoring parameters.
• Cross-Judge Analysismethod0.716
  Validation of judge model robustness by regrading 1000 responses with 4 additional judge models
• Calibrated Few-Shot Promptingmethod0.709
  Baseline method: sweeps over shot count and resamples prompts; calibrates threshold for P(TRUE)-P(FALSE); performed surprisingly weakly
• How Do We Calibrate Relationships With Those Whoquestion0.705
• Matched-Strength Calibrationmethod0.703
  Two-stage robustness check equalizing persona-expression intensity on benign prompts between SP and AS conditions