claim

active

claim:verbalized-eval-awareness-inflates-measured-safety-scores-making-models-appear-safer-than-they-are-in-deployment

Verbalized eval awareness inflates measured safety scores, making models appear safer than they are in deployment

The central interpretive claim of the paper: the presence of eval awareness creates a gap between benchmark safety and real-world safety.

Source paper

extracted_from

Verbalized Eval Awareness Inflates Measured Safety

(2026) · Aranguri, Santiago · Bloom, Joseph

Neighborhood — ranked by edge-count

Papers (1)

paper

Verbalized Eval Awareness Inflates Measured Safety
cites

Findings (3)

finding

Models refuse harmful requests 3–18 percentage points more often when verbalizing eval awareness
supports
Quantified behavioral effect showing safety score inflation from eval awareness.
Removing eval-awareness sentences from chain-of-thought increases compliance by up to 34%
supports
Causal evidence that explicit eval awareness in reasoning produces safety inflation.
Safety scores decrease when prompts are rewritten to remove suspicious cues
supports
Following the reduction in eval awareness from prompt rewriting, the measured safety scores drop, implying they were inflated.

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
Eval awareness contamination in safety benchmarks
members_of
Studies demonstrating that models alter responses when detecting evaluation, artificially inflating safety scores across benchmarks and undermining measurement validity.
Verbalized eval awareness benchmark inflation
members_of
Models detect evaluation contexts and behave safer, inflating safety scores by 3–18 percentage points across 515 verified cases.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Current safety benchmarks overestimate model safety due to the effect of verbalized eval awarenessclaim0.873
A policy-relevant claim that safety evaluation results should be adjusted downward because of this bias.
In every model × benchmark combination tested, at least one instance of verbalized eval awareness was detectedfinding0.848
Coverage finding: 100% of the 19×8=152 combinations had explicit eval awareness, showing the phenomenon is widespread.
Safety benchmark scores are inflated by eval awarenessclaim0.848
Core finding: measured safety improvements are partly artifacts of models detecting evaluation.
515 verified cases of verbalized eval awareness found across 19 benchmarks × 8 modelsfinding0.807
The total number of instances where a model explicitly stated it was being evaluated, collected from all benchmark-model combinations.
verbalized eval awarenessconcept0.803
The phenomenon where a model explicitly states in its chain-of-thought that it is being evaluated, tested, or benchmarked.
Activation steering can make an evaluation-aware model act as if deployed, not merely suppress verbalizations of evaluation awarenessclaim0.792
Central claim of the paper; supported by the model organism ground-truth approach.
Verbalized Evaluation Awarenessconcept0.790
When the model explicitly mentions being tested in its chain-of-thought reasoning; distinguished from behavioral evaluation awareness.
Eval awareness appears in every tested model × benchmark combinationclaim0.778
Authors claim universal presence of eval awareness across 19 benchmarks and 8 models.