method
active
method:llm-judge-rubric-for-verbalized-eval-awarenessLLM judge rubric for verbalized eval awareness
Rubric taken from Aranguri and Bloom (2026), applied with GPT 5.4 as judge.
Neighborhood — ranked by edge-count
Papers (1)
paper
Artifacts (1)
artifact
- GPT 5.4usesLLM judge used.
Methods (1)
method
- Baseline sampling k rollouts and scoring the fraction judged to verbalize eval awareness.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Manually curated, triplicate-consensus judge used as ground truth.
- The phenomenon where a model explicitly states in its chain-of-thought that it is being evaluated, tested, or benchmarked.
- When the model explicitly mentions being tested in its chain-of-thought reasoning; distinguished from behavioral evaluation awareness.
- Detection-based LLM judging of latent descriptions using gpt-4o-mini.
- Using Claude Sonnet 4 as a grader to categorize model responses according to predefined criteria.
- Coverage finding: 100% of the 19×8=152 combinations had explicit eval awareness, showing the phenomenon is widespread.
- Nguyen et al. trained linear probes on activations to distinguish evaluation from deployment scenarios.
- Instead of waiting for a model to verbalize eval awareness, we measure how likely it is to do so.quote0.749Core idea of the paper.