method
active
method:model-diff-amplificationModel diff amplification
Core method: sample from logits_after + α(logits_after − logits_before) to surface rare behaviors.
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (3)
concept
- Model DiffingextendsApproach to identify interpretable differences between a base LLM and its fine-tuned version using SAE latents
- The phenomenon where finetuning on narrow-domain tasks produces broad misalignment extending far beyond the training domain
- Sleeper agentaboutModel trained to behave harmlessly but later exhibits harmful behavior; features may reveal such hidden objectives.
Methods (2)
method
- Iterative sampling procedure recomputing before/after logits with each new token.
- Measures emergent misalignment by counting responses judged harmful by an LLM.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Positioning.
- Prior method extrapolating beyond the original model in logit space to surface rare behaviors.
- Conclusion.
- Scope limitation.
- Problem cited as a limitation of current LLMs; PRH predicts larger models should amplify bias less
- Train a single SAE encoding and decoding activations from two models simultaneously to find shared and exclusive features.
- The paper's primary mechanistic analysis method: comparing SAE latent activations before and after fine-tuning to identify misalignment-relevant features
- Generative models that reverse a noising process, mentioned in quasi-simulator table.