Reinforcement learning with graders rewarding incorrect responses on o3-mini causes emergent misalignment in multiple domains

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Helpful-only o3-mini models show substantially more emergent misalignment than safety-trained models under RLfinding0.803
  In RL (but not SFT) experiments, removal of safety training amplifies misalignment generalization
• During pre-training, models learn a variety of personas including misaligned ones; fine-tuning on narrowly incorrect datasets amplifies misaligned personas because they reduce training loss, causing broadly misaligned behaviorclaim0.792
  The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment
• Emergent misalignment occurs across diverse settings including RL, multiple advice domains, and models without safety trainingfinding0.786
  Section 2 core result establishing generality of emergent misalignment
• Turner et al. 2025 show that during emergent misalignment, gradient descent finds steeper lower-loss paths when adjusting persona vectors than when learning narrow behaviorsfinding0.784
  Mechanistic explanation of why fine-tuning shifts persona vectors rather than directly learning narrow behaviors
• Reinforcement learning acting on individual characteristics affecting their connections to others can result in dynamics that are equivalent to unsupervised learning at the system scale.claim0.783
  Key insight linking individual rewards to system-level learning.
• Generalized misalignment is 'easy to specify' via reinforcement learning, perhaps because it taps into a representation already present in the model from pre-trainingclaim0.776
  RL with only scalar reward induces emergent misalignment, suggesting misalignment is a natural pre-existing representation
• Misaligned models might acquire evaluation awareness through reward hacking or goal misgeneralization during normal training without deliberate designhypothesis0.770
  Motivation for the two-stage training design; links the model organism to plausible natural emergence.
• Reinforcement learning can be regarded as a limiting or special case of model-based approaches in general — or active inference in particular — when epistemic value is removed.claim0.769
  §3 Discussion.