finding
active
finding:pretraining-only-data-without-chat-transcripts-reduced-exclusive-features-by-20-but-did-not-produce-larger-asymmetryPretraining-only data without chat transcripts reduced exclusive features by ~20% but did not produce larger asymmetry
Tests the chat oversampling hypothesis.
Source paper
extracted_fromSiddharth Mishra-Sharma · Trenton Bricken · Jack Lindsey · Adam Jermyn +4
Neighborhood — ranked by edge-count
Papers (1)
paper
Hypotheses (1)
hypothesis
- Similar number and nature of exclusive features across models results from oversampling of Human/Assistant chat transcriptsassociated_withcontradictsMotivated by Bricken et al; tested and not supported.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- From Lindsey et al; modeled with co-activation toy.
- SAE latents depend on training data.
- The shape of the pretraining corpus is a direct lever on which traits a base model can expresshypothesis0.757Forward-looking hypothesis about pretraining data as mechanism for persona formation
- Practical safety question about intervening on persona representations during pretraining
- E.g. 'Here is a draft', apology, and 'explain like I'm in sixth grade' features.
- Reveals that distance-only loss undesirably decreases similarity to positive centroid alongside negative
- The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment
- Curating pretraining data to emphasize healthy emotional regulation may shape the model's emotional foundations beneficiallyhypothesis0.739Proposed intervention.