thinker
active
thinker:nicholas-l-turner

Nicholas L. Turner

Authored
2
Introduces
0
Studies
0
Affiliations
1
Cited by
8

Authored papers (2)

  • Interference weights are measurable inside a trained transformer, and the model keeps its most effective virtual weights helpful while confining harmful ones to a low-effectiveness range. In a one-layer, 2.9M-parameter model (d_m=256, 4,096-token vocab) expanded into a 331M-weight virtual weight (VW) model of tokens, positions, transcoder features and logits, Fisher effectiveness (a second-order KL estimate in nats) paired with helpfulness (the loss change under ablation) separates function from noise. The largest IN→"utions" Tokens→Logits weight never fires correctly in training, and its effectiveness sits roughly three orders of magnitude below the best weights from the same token. Raw virtual weight magnitude is a weak guide: effectiveness varies over about 10 orders of magnitude, and the median weight is 10,000× below the maximum. Pruning the least effective 70% costs about 0.01 nats and 85% costs under 0.1, yet the VW model stays dense. Helpfulness sampling finds 47.6% of weights with positive mean helpfulness, and preserving 99% of helpfulness mass needs 13.6% density. That is still tens of millions of weights, more than the original parameter count. Effectiveness is necessary but not sufficient for reading global circuits. Removing interference in the token and feature basis will not yield a sparse model. Progress depends on finding a better basis, with these metrics serving as the test for recognizing one.

More papers — OpenAlex / S2

Affiliations (1)

Co-authors (12)

Other inbound relations (1)

Recent mentions (2)