claim
active
claim:exclusive-features-become-dense-and-polysemantic-because-they-compete-with-shared-features-for-limited-feature-capacity-and-must-encode-more-to-justify-their-allocationExclusive features become dense and polysemantic because they compete with shared features for limited feature capacity and must encode more to justify their allocation
Central interpretation.
Source paper
extracted_fromSiddharth Mishra-Sharma · Trenton Bricken · Jack Lindsey · Adam Jermyn +4
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (3)
finding
- With 1024 learnable features and 700 true features (500 shared, 100 exclusive per model), exclusive features take on higher densities in the toy modelassociated_withsupportsToy model in the limited-capacity regime reproduces the density effect.
- With 4096 learnable features and 450 true features (300 shared, 75 exclusive per model), no density contrast between shared and exclusive features appearsassociated_withsupportsToy model in the abundant-capacity regime.
- Primary unexpected empirical observation.
Claims (2)
claim
- Mechanistic argument.
- Links toy model to practice.
Methods (1)
method
- Toy model of crosscoder diffingsupportsSynthetic two-model activations built from shared and exclusive latent factors with controllable ground truth.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Main proposal.
- Major prediction checked against InceptionV1.
- As we make a model wider, do we just get more features, or the same features packed less densely?question0.776Fresh-traction question.
- Central claim of the paper, supported by detailed feature analysis, human evaluation, automated interpretability of activations, and automated interpretability of logit weights
- Mitigation works on real models.
- E.g. 'Here is a draft', apology, and 'explain like I'm in sixth grade' features.
- Third empirical pattern.
- Motivation for DFC.