concept
active
concept:model-diffingModel Diffing
Approach to identify interpretable differences between a base LLM and its fine-tuned version using SAE latents
Neighborhood — ranked by edge-count
Papers (4)
paper
- Insights on Crosscoder Model Diffingextendsmentions
Thinkers (3)
thinker
- Roger GrossestudiesUnpublished independent model diffing work.
- Shah et al.studiesAuthors of persona modulation jailbreak dataset used in this paper's evaluations
- Kissane et al.studiesShowed base-model SAEs transfer to finetuned models.
Claims (1)
claim
- Authors' outlook.
Methods (3)
method
- Sparse CrosscoderimplementsSparse dictionary learning variant that reads and writes to multiple layers or models, yielding shared features.
- Crosscoder model diffingextendsTrain a single SAE encoding and decoding activations from two models simultaneously to find shared and exclusive features.
- Model diff amplificationextendsCore method: sample from logits_after + α(logits_after − logits_before) to surface rare behaviors.
Concepts (2)
concept
- Extension of model diffing to models with different architectures.
- Unknown unknownsassociated_withNovel behavioral tendencies not covered by existing evaluation suites.
Institutes (1)
institute
- OpenAI Clarity TeamintroducesOriginally articulated model diffing in 2018.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- The paper's primary mechanistic analysis method: comparing SAE latent activations before and after fine-tuning to identify misalignment-relevant features
- Injecting the activation difference between fine-tuned and original model into the AO.
- Core motivation.
- Synthetic two-model activations built from shared and exclusive latent factors with controllable ground truth.
- The phenomenon of model internals deviating from desired behavior; MAS is demonstrated to detect this via comparison of toxic vs nontoxic LLMs.
- LLM behavior of generating falsehoods; the multi-dimensional truth subspace raises new risks for subtle manipulation
- Conclusion.
- When a single conversation involves different models, e.g., routing between standard and reasoning models