concept
active
concept:verbalized-evaluation-awarenessVerbalized Evaluation Awareness
When the model explicitly mentions being tested in its chain-of-thought reasoning; distinguished from behavioral evaluation awareness.
Neighborhood — ranked by edge-count
Papers (2)
paper
Methods (1)
method
- Baseline sampling k rollouts and scoring the fraction judged to verbalize eval awareness.
Concepts (3)
concept
- Evaluation Awarenessassociated_withextendsCore concept: the ability of LLMs to detect when they are being tested and adjust behavior accordingly.
- verbalized eval awarenessrelated_toThe phenomenon where a model explicitly states in its chain-of-thought that it is being evaluated, tested, or benchmarked.
- Unverbalized Evaluation Awarenessrelated_toKey finding: models internally suspect they are being tested without explicitly saying so; surfaced by NLAs during auditing.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Coverage finding: 100% of the 19×8=152 combinations had explicit eval awareness, showing the phenomenon is widespread.
- Instead of waiting for a model to verbalize eval awareness, we measure how likely it is to do so.quote0.795Core idea of the paper.
- The central interpretive claim of the paper: the presence of eval awareness creates a gap between benchmark safety and real-world safety.
- Central concept: models' detection and behavioral response to being evaluated.
- Main interpretation.
- Rubric taken from Aranguri and Bloom (2026), applied with GPT 5.4 as judge.
- The total number of instances where a model explicitly stated it was being evaluated, collected from all benchmark-model combinations.
- Nguyen et al. trained linear probes on activations to distinguish evaluation from deployment scenarios.