finding
active
finding:most-effective-weights-are-helpful-exceeding-any-harmful-weight-by-an-order-of-magnitude-within-each-family-1-111-weight-sampleMost effective weights are helpful, exceeding any harmful weight by an order of magnitude within each family (1,111-weight sample)
Three regimes by effectiveness.
Source paper
extracted_fromNicholas L. Turner · Jeffrey Wu · Joshua Batson
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (2)
claim
- Interpretation of effectiveness tail result.
- Reads loss change directly.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Helpful and harmful weights are scattered across the entire range of virtual weight magnitudesfinding0.834IN Tokens→Logits shows weak VW-helpfulness relationship.
- Estimates density floor.
- Quantitative constraint on the levels of scale property: centers are most helpful when size ratios are moderate; centers less than one-tenth the size of a larger one are less likely to help it
- Interpretation of motivating example.
- Large virtual weights are not inherently misleading but not guaranteed to be helpful or effectiveclaim0.762Caution for practitioners.
- Effectiveness doesn't fully clean.
- Discussion.
- Implication of PRH for 'scale is all you need' argument