Turner et al. 2025 show that during emergent misalignment, gradient descent finds steeper lower-loss paths when adjusting persona vectors than when learning narrow behaviors

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• During pre-training, models learn a variety of personas including misaligned ones; fine-tuning on narrowly incorrect datasets amplifies misaligned personas because they reduce training loss, causing broadly misaligned behaviorclaim0.823
  The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment
• Emergent misalignment is easy to reverse with narrow fine-tuning on benign data because it is an instance of surprisingly strong misalignment generalization that works bidirectionallyclaim0.816
  The emergent realignment result showing ~120 samples reverse full misalignment supports this interpretive claim
• Emergent misalignment appears when between 25% to 75% of the fine-tuning dataset consists of incorrect data (health: 25%, code: 75%)finding0.800
  Dataset mixture experiments establish the fraction of incorrect data needed to induce misalignment
• The directional transition pattern of persona vectors at the identified attention layer represents a unique footprint of persona representations, distinct from general hidden state dynamicsclaim0.788
  Supported by comparing persona vector transitions to hidden vector transitions from OpenAssistant data
• Misaligned models might acquire evaluation awareness through reward hacking or goal misgeneralization during normal training without deliberate designhypothesis0.784
  Motivation for the two-stage training design; links the model organism to plausible natural emergence.
• The behavior of the initial model may be more impactful in determining emergent misalignment for on-policy methods (RL) than for off-policy methods (SFT)claim0.784
  RL shows stronger safety training effect while SFT does not, suggesting on-policy methods are more sensitive to initial model state
• Models fine-tuned on different narrow datasets (bad medical advice and extreme sports) end up in highly correlated misalignment directions, converging on the same evil persona region (Soligo et al. 2025)finding0.784
  Evidence for the evil persona as a privileged basin supporting Hypothesis 3
• Reinforcement learning with graders rewarding incorrect responses on o3-mini causes emergent misalignment in multiple domainsfinding0.784
  First demonstration that RL-induced misalignment (not just SFT) produces broadly misaligned behavior