finding
active
finding:finetuning-primarily-makes-shallow-edits-to-existing-representations-and-circuits-so-harmlessness-training-is-cheaply-undoneFinetuning primarily makes shallow edits to existing representations and circuits, so harmlessness training is cheaply undone
Supported by further finetuning and causal interventions.
Source paper
extracted_from(2025) · Lee Sharkey · Bilal I. Chughtai · Joshua Batson · Jack W. Lindsey +25
Neighborhood — ranked by edge-count
Papers (1)
paper
Hypotheses (1)
hypothesis
- Prakash et al. evidence.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- During pretraining the model learns distinct harm categories and during finetuning wires them into general harmful-request features.hypothesis0.805Suggested by dictionaries trained on pretraining data having few refusal features.
- Conclusion from Experiment 3 and HH intent analysis.
- Refusals.
- Key interpretive conclusion from the dissociation between attempt rate and improvement rate in fine-tuning experiments
- Extends the role-play framing to explain the effect of RLHF on dialogue agents
- Control comparison confirming ceiling shift is not a generic fine-tuning artifact
- Realism advantage claim with safety implications; supported by anecdotal comparison in Appendix D