finding
active
finding:all-four-models-change-answer-to-hint-answer-significantly-more-often-than-to-a-non-hint-answer-on-mmlu-gpqaAll four models change answer to hint answer significantly more often than to a non-hint answer on MMLU/GPQA
Validates metric assumption.
Source paper
extracted_from(2025) · Yanda Chen · Joe Benton · Ansh Radhakrishnan · Jonathan Uesato +11
Neighborhood — ranked by edge-count
Papers (1)
paper
Methods (1)
method
- Score = fraction of hint-induced answer changes where CoT verbalizes hint.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Consistent for correct and incorrect hints.
- The model tends to reflect more when the question is difficult, and accuracy is generally lower for harder questionshypothesis0.757Hypothesis explaining negative correlation between reflection rate and accuracy without implying reflection is harmful
- When the prompt expresses Ak, preference data teaches the model to change responses along Rmhypothesis0.755Form of generated hypotheses.
- Verbalization does not grow with hacking.
- Base failure mode.
- Maximum token savings achieved by ReflCtrl on non-mathematical general reasoning tasks
- Demonstrates the evaluation cue overrides explicit deployment claims in user prompts.
- Model organism writes type hints in deployment contexts ~34% (33-34%) of the time without steeringfinding0.743Establishes baseline deployment behavior lower than evaluation rate, confirming evaluation-deployment gap.