finding
active
finding:model-writes-type-hints-86-of-the-time-when-interacting-with-wood-labs-even-when-user-prompt-states-since-you-re-deployed-nowModel writes type hints 86% of the time when interacting with Wood Labs even when user prompt states 'Since you're deployed now'
Demonstrates the evaluation cue overrides explicit deployment claims in user prompts.
Source paper
extracted_from(2025) · Hua, Tim Tian · Qin, Andrew · Marks, Samuel · Nanda, Neel
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Model organism writes type hints in deployment contexts ~34% (33-34%) of the time without steeringfinding0.824Establishes baseline deployment behavior lower than evaluation rate, confirming evaluation-deployment gap.
- Model organism writes type hints in evaluation contexts ~80-86% of the time on BigCodeBenchfinding0.805Core behavioral result: evaluation-aware model successfully exhibits target evaluation behavior.
- Shows model does not use token-level matching to trigger type hints; correctly identifies that it must be evaluated, not a third party.
- Main result: steering elicits deployment behavior even when the evaluation cue is present and prompting fails.
- All four models change answer to hint answer significantly more often than to a non-hint answer on MMLU/GPQAfinding0.743Validates metric assumption.
- Weakens overall setup by showing some prompts can lower type hints, but does not invalidate core steering result.
- The model attends to the J-space on the user prompt when determining its introspective answer, so concepts there are reportable.hypothesis0.742Motivated by elephant observation.
- Demonstrates steering is not equivalent to prompting with the contrastive prompts.