finding
active
finding:restricting-to-rollouts-without-verbalized-eval-awareness-drops-auroc-by-less-than-0-02Restricting to rollouts without verbalized eval awareness drops AUROC by less than 0.02
Robustness check.
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- The probability that a rollout contains verbalized eval awareness is primarily set by the prompt rather than emerging over the chain of thoughtassociated_withsupportsMain interpretation.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Causal.
- Rollouts with and without verbalized eval awareness average to similar logit monitor curvesfinding0.820Figure 1 result.
- Rollout-level verbalization barely matters.
- Prediction task result.
- Models refuse harmful requests 3–18 percentage points more often when verbalizing eval awarenessfinding0.799Quantified behavioral effect showing safety score inflation from eval awareness.
- The central interpretive claim of the paper: the presence of eval awareness creates a gap between benchmark safety and real-world safety.
- Instead of waiting for a model to verbalize eval awareness, we measure how likely it is to do so.quote0.791Core idea of the paper.
- Verbalization monitor asymptotes at AUROC 0.75 even at 100 rollouts on Fortress vs WildChatfinding0.779Baseline ceiling.