finding
active
finding:model-exclusive-features-have-feature-densities-about-an-order-of-magnitude-higher-than-shared-features-in-base-vs-helpful-only-claude-3-sonnet-like-diffsModel-exclusive features have feature densities about an order of magnitude higher than shared features in base vs helpful-only Claude 3 Sonnet-like diffs
Primary unexpected empirical observation.
Source paper
extracted_fromSiddharth Mishra-Sharma · Trenton Bricken · Jack Lindsey · Adam Jermyn +4
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Central interpretation.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Near-identical numbers of exclusive features are found for both models in base vs finetuned diffsfinding0.821Symmetry of exclusive features across models.
- E.g. 'Here is a draft', apology, and 'explain like I'm in sixth grade' features.
- Authors take agnostic position on ontological status but universality evidence pushes toward features being real
- Remains incompletely understood.
- Mixed picture for diffing.
- Similar number and nature of exclusive features across models results from oversampling of Human/Assistant chat transcriptshypothesis0.797Motivated by Bricken et al; tested and not supported.
- Limitation.
- Features writing predominantly to one model.