Eval Awareness

Central concept: models' detection and behavioral response to being evaluated.

Neighborhood — ranked by edge-count

paper

community

concept

verbalized eval awareness
extendsrelated_to
The phenomenon where a model explicitly states in its chain-of-thought that it is being evaluated, tested, or benchmarked.
Evaluation Awareness
related_to
Core concept: the ability of LLMs to detect when they are being tested and adjust behavior accordingly.
Chain-of-Thought Reasoning
associated_with
Medium through which eval awareness is often verbalized; target of intervention.
steering vectors
associated_with
A method for modifying model behavior by adding perturbation vectors to activations, used here to try to reduce eval awareness.
prompt rewriting to remove suspicious cues
associated_with
A technique used in the paper to alter prompts so they contain fewer hints that the interaction is a safety evaluation.
safety scores
associated_with
Metrics derived from benchmarks to quantify how safe a model is, e.g., refusal rate to harmful requests.
Safety benchmarks
associated_with
Evaluation framework whose validity is questioned by presence of eval awareness.

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

eval() Operationmethod0.829
Linda primitive that creates a live tuple (new process); it turns into a data tuple upon termination.
Eval awareness appears in every tested model × benchmark combinationclaim0.808
Authors claim universal presence of eval awareness across 19 benchmarks and 8 models.
Unverbalized Evaluation Awarenessconcept0.795
Key finding: models internally suspect they are being tested without explicitly saying so; surfaced by NLAs during auditing.
expanded awarenessconcept0.790
Wide attentional radius with all-to-all correlation, associated with Claude models; enables better self-monitoring and alignment.
Verbalized Evaluation Awarenessconcept0.783
When the model explicitly mentions being tested in its chain-of-thought reasoning; distinguished from behavioral evaluation awareness.
Open awarenessconcept0.774
Meditative state of feeling the undoctored hum of SOHMs without grasping or pushing away.
Emergence Of Awarenessconcept0.774
Safety benchmark scores are inflated by eval awarenessclaim0.763
Core finding: measured safety improvements are partly artifacts of models detecting evaluation.