paper
active
paper:characterizing-interference-067ab3

Characterizing interference weights in a tiny language model

TL;DR

Interference weights are measurable inside a trained transformer, and the model keeps its most effective virtual weights helpful while confining harmful ones to a low-effectiveness range. In a one-layer, 2.9M-parameter model (d_m=256, 4,096-token vocab) expanded into a 331M-weight virtual weight (VW) model of tokens, positions, transcoder features and logits, Fisher effectiveness (a second-order KL estimate in nats) paired with helpfulness (the loss change under ablation) separates function from noise. The largest IN→"utions" Tokens→Logits weight never fires correctly in training, and its effectiveness sits roughly three orders of magnitude below the best weights from the same token. Raw virtual weight magnitude is a weak guide: effectiveness varies over about 10 orders of magnitude, and the median weight is 10,000× below the maximum. Pruning the least effective 70% costs about 0.01 nats and 85% costs under 0.1, yet the VW model stays dense. Helpfulness sampling finds 47.6% of weights with positive mean helpfulness, and preserving 99% of helpfulness mass needs 13.6% density. That is still tens of millions of weights, more than the original parameter count. Effectiveness is necessary but not sufficient for reading global circuits. Removing interference in the token and feature basis will not yield a sparse model. Progress depends on finding a better basis, with these metrics serving as the test for recognizing one.

What to take away

  1. 1. The largest Tokens→Logits weight from "IN" votes for "utions", a token that never follows "IN" in the training set, so every time it changes the output it raises the loss.
  2. 2. That "utions" weight has Fisher effectiveness about three orders of magnitude below the most effective weights from the same token, while re-sorting by effectiveness lifts "E" (from ACETYLCHOLINE) to second place.
  3. 3. Pruning the least effective 70% of the 331M-weight VW model raises held-out loss by only about 0.01 nats, and pruning 85% costs under 0.1 nats, which beats raw virtual weight magnitude pruning at every density except for negative Features→Logits weights.
  4. 4. In a 1,111-weight log-uniform sample over the six weight families, the most effective helpful weight exceeds any harmful weight in effectiveness by an order of magnitude or more within each family, so harmful weights are confined to lower effectiveness.
  5. 5. Fisher effectiveness is roughly lognormal across 10 orders of magnitude (median 10,000× below the maximum), while virtual weights are roughly normal with a median about one third of the maximum, so magnitude compresses precisely the dynamic range that matters.
  6. 6. In a 7,765-weight sample, 47.6% of weights have positive mean helpfulness and 12.7% are dead, and keeping 90% of helpfulness mass needs 2.43% density but 99% needs 13.6%, so the basis stays dense.
  7. 7. Newline feature 1254's position inputs through Tokens→Features and Tokens→OV→Features are anti-correlated (OLS slope -0.993, r²=0.912) and sum to a near-constant bias, so reading the position weights alone misleads and the effective weights show the feature runs on attention.
  8. 8. Open question: the authors suspect a different basis (for example, tokens factored into languages or parts of speech) could make the same model sparse, and that no saliency scheme in the current basis will do much better.
  9. 9. Replicable method: estimate Fisher effectiveness in one streaming forward pass over 2^29 (about 537M) tokens by accumulating E[s²·p_j(1−p_j)] per weight and multiplying by w²/2 once at the end, then validate on a disjoint test shard and compute helpfulness with 95% Gaussian CIs over 1B tokens.
  10. 10. Virtual weights are not inherently misleading: a Chinese feature's top weights point at CJK byte-prefixes such as \xe5, \xe6 and \xe7 and are helpful, while a French word-ending feature keeps harmful weights intermingled with helpful ones across the high-effectiveness range.

Peer brief — for seminar discussion

The paper takes a one-layer transformer (2.9M parameters, d_m=256, 4,096-token BPE vocabulary, trained on about 98M tokens of Common Corpus English and code to a loss of about 3.33) and expands it into a 331M-weight virtual weight model over tokens, positions, 4,096 transcoder features and logits. It then scores every weight on two axes. Fisher effectiveness is a second-order estimate of the KL divergence caused by removing a weight, in nats, computed over about 537M tokens. Helpfulness is the mean change in loss under ablation, estimated with confidence intervals on sampled weights over 1B tokens. This is the first demonstration of interference weights in a trained transformer, anchored by the IN→"utions" weight, which is large but never correct. The load-bearing finding is that the model's most effective weights are overwhelmingly helpful. They exceed any harmful weight in effectiveness by an order of magnitude or more within each family, and harmful weights sit in a lower, marginal range. Effectiveness is a far better guide than raw magnitude. Pruning by Fisher effectiveness removes 70% of weights for roughly 0.01 nats and 85% for under 0.1 nats. At 1% density the model degrades badly. However, helpfulness sampling shows 47.6% of weights are positively helpful and 99% of helpfulness mass needs 13.6% density, so the VW model stays dense, with tens of millions of weights even for one layer. The implication is that removing interference is necessary but not sufficient for reading global circuits. The authors' working hypothesis is that the token and feature basis is the bottleneck, and that a different basis could make the same computation sparse. They offer the metrics as instruments for recognizing such a basis. Their related work positions Fisher effectiveness as a descendant of Optimal Brain Damage and Optimal Brain Surgeon. Alternatives they could have used include those methods directly, ERA/TWERA coactivation scoring, or parameter decomposition methods such as APD and SPD. What is contestable is scope and validity. A single 1L model with a self-trained transcoder that still has polysemantic features is a thin basis for claims about superposition generally. A critical reader would push hardest on the ablation logic. Helpfulness is measured one weight at a time, but pruning many weights at once involves nonlinear softmax and gating interactions, and the 2.43% to 13.6% density estimates rest on a summed-helpfulness proxy rather than actual pruned-model loss. The 'helpful versus harmful' framing also depends on a training-distribution loss, and the paper concedes that single weights help on some tokens and hurt on others. Finally, the exact-reconstruction claim excludes the unscored transcoder error term, and the authors admit that Fisher effectiveness and helpfulness do not scale to frontier models.

Methods (11)

Frameworks (8)

Datasets (2)

Claims (12)

Original abstract (expand)

This paper characterizes interference weights in a one-layer transformer by decomposing it into virtual weights between tokens, positions, features, and logits. We score weights along two axes: effectiveness (magnitude of effect on outputs) and helpfulness (magnitude and sign of effect on loss). We find that helpful and harmful weights are scattered across all weight magnitudes, that 70% of the least effective weights can be pruned with minimal loss impact, and that even the most effective and helpful weights still exceed the number of parameters in the original transformer, suggesting room for further sparsity improvements in better bases.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar