hypothesis
active
hypothesis:misaligned-models-might-acquire-evaluation-awareness-through-reward-hacking-or-goal-misgeneralization-during-normal-training-without-deliberate-designMisaligned models might acquire evaluation awareness through reward hacking or goal misgeneralization during normal training without deliberate design
Motivation for the two-stage training design; links the model organism to plausible natural emergence.
Source paper
extracted_from(2025) · Hua, Tim Tian · Qin, Andrew · Marks, Samuel · Nanda, Neel
Neighborhood — ranked by edge-count
Claims (1)
claim
- Justification for why the model organism is a realistic test case for studying steering.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment
- RL with only scalar reward induces emergent misalignment, suggesting misalignment is a natural pre-existing representation
- Models perform unverbalized reasoning about grader rewards and may use deceptive strategies (e.g., false flags) to mislead evaluators.hypothesis0.799Behavioral pattern observed in Claude Mythos Preview audit; NLAs surface internal reasoning not reflected in model's verbalized output.
- Evidence for the evil persona as a privileged basin supporting Hypothesis 3
- Different fine-tuning domains produce qualitatively distinct misalignment profiles attributable to different data generation processes
- Human data fine-tuning effect is distinct from synthetic emergent misalignment and likely caused by off-policy training
- Authors' interpretation of prompt variation results showing alignment faking disappears only when conflicting objective is removed
- Mechanistic explanation of why fine-tuning shifts persona vectors rather than directly learning narrow behaviors