hypothesis
active
hypothesis:chatbot-features-exclusive-to-the-pretrained-model-may-have-been-tweaked-in-finetuning-appearing-exclusive-but-being-shared-with-different-contextChatbot features exclusive to the pretrained model may have been tweaked in finetuning, appearing exclusive but being shared with different context
From Lindsey et al; modeled with co-activation toy.
Source paper
extracted_fromSiddharth Mishra-Sharma · Trenton Bricken · Jack Lindsey · Adam Jermyn +4
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Plausible explanation.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Similar number and nature of exclusive features across models results from oversampling of Human/Assistant chat transcriptshypothesis0.808Motivated by Bricken et al; tested and not supported.
- Tests the chat oversampling hypothesis.
- During pretraining the model learns distinct harm categories and during finetuning wires them into general harmful-request features.hypothesis0.796Suggested by dictionaries trained on pretraining data having few refusal features.
- E.g. 'Here is a draft', apology, and 'explain like I'm in sixth grade' features.
- Chat-trained SAEs find behaviourally relevant features because many causal features occur on chat-specific special tokenshypothesis0.763Explanation for LMSYS superiority.
- Refusals.
- Near-identical numbers of exclusive features are found for both models in base vs finetuned diffsfinding0.759Symmetry of exclusive features across models.
- The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment