concept
active
concept:emergent-misalignmentEmergent Misalignment
The phenomenon where finetuning on narrow-domain tasks produces broad misalignment extending far beyond the training domain
Neighborhood — ranked by edge-count
Papers (8)
paper
Methods (1)
method
- Core method: sample from logits_after + α(logits_after − logits_before) to surface rare behaviors.
Concepts (2)
concept
- Emergent Re-alignmentrelated_toThe phenomenon where fine-tuning on small amounts of benign data efficiently reverses emergent misalignment
- In-context emergent misalignmentrelated_toCarefully crafted conversations pushing a post-trained model into the evil persona region without any fine-tuning
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Task of discovering a fine-tuned model has become emergently misaligned, scored on a 0-5 rubric.
- The emergent realignment result showing ~120 samples reverse full misalignment supports this interpretive claim
- The broader phenomenon of misaligned behaviors generalizing beyond the fine-tuning distribution
- Dataset mixture experiments establish the fraction of incorrect data needed to induce misalignment
- Section 2 core result establishing generality of emergent misalignment
- Blackmail scenarios from Lynch et al.
- Alignment faking appears almost exclusively in models at scale of Claude 3 Opus and Claude 3.5 Sonnet
- A multi-dimensional characterization of a model's misaligned behaviors across different behavioral categories