hypothesis
active
hypothesis:similar-number-and-nature-of-exclusive-features-across-models-results-from-oversampling-of-human-assistant-chat-transcriptsSimilar number and nature of exclusive features across models results from oversampling of Human/Assistant chat transcripts
Motivated by Bricken et al; tested and not supported.
Source paper
extracted_fromSiddharth Mishra-Sharma · Trenton Bricken · Jack Lindsey · Adam Jermyn +4
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (1)
finding
- Pretraining-only data without chat transcripts reduced exclusive features by ~20% but did not produce larger asymmetryassociated_withcontradictsTests the chat oversampling hypothesis.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- From Lindsey et al; modeled with co-activation toy.
- Primary unexpected empirical observation.
- E.g. 'Here is a draft', apology, and 'explain like I'm in sixth grade' features.
- Mixed picture for diffing.
- Near-identical numbers of exclusive features are found for both models in base vs finetuned diffsfinding0.774Symmetry of exclusive features across models.
- Frequently repeated chat features (e.g. knowledge cutoff) are memorized by the SAE and base model, causing isolationhypothesis0.771Second explanation for isolated features.
- Authors take agnostic position on ontological status but universality evidence pushes toward features being real
- Empirical characterization of conversation domains that are safe for model persona stability