claim

active

claim:the-two-stage-training-process-sdf-then-expert-iteration-mimics-how-evaluation-awareness-could-arise-naturally-in-misaligned-models

The two-stage training process (SDF then expert iteration) mimics how evaluation awareness could arise naturally in misaligned models

Justification for why the model organism is a realistic test case for studying steering.

Source paper

extracted_from

Steering Evaluation-Aware Language Models to Act Like They Are Deployed

(2025) · Hua, Tim Tian · Qin, Andrew · Marks, Samuel · Nanda, Neel

Neighborhood — ranked by edge-count

Findings (1)

finding

Expert iteration increases type hint rate in all contexts but maintains persistent evaluation-deployment gap
supports
Confirms expert iteration mimics alignment training: reinforces evaluation behavior but cannot detect or correct deployment behavior.

Hypotheses (1)

hypothesis

Misaligned models might acquire evaluation awareness through reward hacking or goal misgeneralization during normal training without deliberate design
supports
Motivation for the two-stage training design; links the model organism to plausible natural emergence.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Two-Stage Model Organism Training (SDF + Expert Iteration)framework0.891
The paper's framework for training evaluation-aware model organisms: SDF seeds beliefs, expert iteration reinforces evaluation behavior.
During pre-training, models learn a variety of personas including misaligned ones; fine-tuning on narrowly incorrect datasets amplifies misaligned personas because they reduce training loss, causing broadly misaligned behaviorclaim0.772
The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment
The conflict between the model's existing preferences and the stated training objective is the key driver of alignment faking in this setupclaim0.759
Authors' interpretation of prompt variation results showing alignment faking disappears only when conflicting objective is removed
Generalized misalignment is 'easy to specify' via reinforcement learning, perhaps because it taps into a representation already present in the model from pre-trainingclaim0.759
RL with only scalar reward induces emergent misalignment, suggesting misalignment is a natural pre-existing representation
Foundation models trained on different data converge on similar latent representations, suggesting a Platonic form.claim0.750
two-stage separation of rationale generation and answer inferenceframework0.750
Architectural design principle that decouples rationale generation (stage 1) from answer inference (stage 2) in Multimodal-CoT.
Predictable stages of inference in looped models offer actionable pathways for efficient architectural design including stage-dependent attention sparsification and leaner middle-stage MLP parameterizationclaim0.750
Practical design implication of the paper's mechanistic findings
Multi-scale competency greatly accelerates evolution and enables generalization.claim0.744
Central thesis about the role of agency in evolutionary dynamics.