paper:characterizing-interference-067ab3Characterizing interference weights in a tiny language model
TL;DR
Interference weights are measurable inside a trained transformer, and the model keeps its most effective virtual weights helpful while confining harmful ones to a low-effectiveness range. In a one-layer, 2.9M-parameter model (d_m=256, 4,096-token vocab) expanded into a 331M-weight virtual weight (VW) model of tokens, positions, transcoder features and logits, Fisher effectiveness (a second-order KL estimate in nats) paired with helpfulness (the loss change under ablation) separates function from noise. The largest IN→"utions" Tokens→Logits weight never fires correctly in training, and its effectiveness sits roughly three orders of magnitude below the best weights from the same token. Raw virtual weight magnitude is a weak guide: effectiveness varies over about 10 orders of magnitude, and the median weight is 10,000× below the maximum. Pruning the least effective 70% costs about 0.01 nats and 85% costs under 0.1, yet the VW model stays dense. Helpfulness sampling finds 47.6% of weights with positive mean helpfulness, and preserving 99% of helpfulness mass needs 13.6% density. That is still tens of millions of weights, more than the original parameter count. Effectiveness is necessary but not sufficient for reading global circuits. Removing interference in the token and feature basis will not yield a sparse model. Progress depends on finding a better basis, with these metrics serving as the test for recognizing one.
What to take away
- 1. The largest Tokens→Logits weight from "IN" votes for "utions", a token that never follows "IN" in the training set, so every time it changes the output it raises the loss.
- 2. That "utions" weight has Fisher effectiveness about three orders of magnitude below the most effective weights from the same token, while re-sorting by effectiveness lifts "E" (from ACETYLCHOLINE) to second place.
- 3. Pruning the least effective 70% of the 331M-weight VW model raises held-out loss by only about 0.01 nats, and pruning 85% costs under 0.1 nats, which beats raw virtual weight magnitude pruning at every density except for negative Features→Logits weights.
- 4. In a 1,111-weight log-uniform sample over the six weight families, the most effective helpful weight exceeds any harmful weight in effectiveness by an order of magnitude or more within each family, so harmful weights are confined to lower effectiveness.
- 5. Fisher effectiveness is roughly lognormal across 10 orders of magnitude (median 10,000× below the maximum), while virtual weights are roughly normal with a median about one third of the maximum, so magnitude compresses precisely the dynamic range that matters.
- 6. In a 7,765-weight sample, 47.6% of weights have positive mean helpfulness and 12.7% are dead, and keeping 90% of helpfulness mass needs 2.43% density but 99% needs 13.6%, so the basis stays dense.
- 7. Newline feature 1254's position inputs through Tokens→Features and Tokens→OV→Features are anti-correlated (OLS slope -0.993, r²=0.912) and sum to a near-constant bias, so reading the position weights alone misleads and the effective weights show the feature runs on attention.
- 8. Open question: the authors suspect a different basis (for example, tokens factored into languages or parts of speech) could make the same model sparse, and that no saliency scheme in the current basis will do much better.
- 9. Replicable method: estimate Fisher effectiveness in one streaming forward pass over 2^29 (about 537M) tokens by accumulating E[s²·p_j(1−p_j)] per weight and multiplying by w²/2 once at the end, then validate on a disjoint test shard and compute helpfulness with 95% Gaussian CIs over 1B tokens.
- 10. Virtual weights are not inherently misleading: a Chinese feature's top weights point at CJK byte-prefixes such as \xe5, \xe6 and \xe7 and are helpful, while a French word-ending feature keeps harmful weights intermingled with helpful ones across the high-effectiveness range.
Peer brief — for seminar discussion
The paper takes a one-layer transformer (2.9M parameters, d_m=256, 4,096-token BPE vocabulary, trained on about 98M tokens of Common Corpus English and code to a loss of about 3.33) and expands it into a 331M-weight virtual weight model over tokens, positions, 4,096 transcoder features and logits. It then scores every weight on two axes. Fisher effectiveness is a second-order estimate of the KL divergence caused by removing a weight, in nats, computed over about 537M tokens. Helpfulness is the mean change in loss under ablation, estimated with confidence intervals on sampled weights over 1B tokens. This is the first demonstration of interference weights in a trained transformer, anchored by the IN→"utions" weight, which is large but never correct. The load-bearing finding is that the model's most effective weights are overwhelmingly helpful. They exceed any harmful weight in effectiveness by an order of magnitude or more within each family, and harmful weights sit in a lower, marginal range. Effectiveness is a far better guide than raw magnitude. Pruning by Fisher effectiveness removes 70% of weights for roughly 0.01 nats and 85% for under 0.1 nats. At 1% density the model degrades badly. However, helpfulness sampling shows 47.6% of weights are positively helpful and 99% of helpfulness mass needs 13.6% density, so the VW model stays dense, with tens of millions of weights even for one layer. The implication is that removing interference is necessary but not sufficient for reading global circuits. The authors' working hypothesis is that the token and feature basis is the bottleneck, and that a different basis could make the same computation sparse. They offer the metrics as instruments for recognizing such a basis. Their related work positions Fisher effectiveness as a descendant of Optimal Brain Damage and Optimal Brain Surgeon. Alternatives they could have used include those methods directly, ERA/TWERA coactivation scoring, or parameter decomposition methods such as APD and SPD. What is contestable is scope and validity. A single 1L model with a self-trained transcoder that still has polysemantic features is a thin basis for claims about superposition generally. A critical reader would push hardest on the ablation logic. Helpfulness is measured one weight at a time, but pruning many weights at once involves nonlinear softmax and gating interactions, and the 2.43% to 13.6% density estimates rest on a summed-helpfulness proxy rather than actual pruned-model loss. The 'helpful versus harmful' framing also depends on a training-distribution loss, and the paper concedes that single weights help on some tokens and hurt on others. Finally, the exact-reconstruction claim excludes the unscored transcoder error term, and the authors admit that Fisher effectiveness and helpfulness do not scale to frontier models.
Methods (11)
- Counterfactual Δf attributionComputes feature attribution passing ablated pre-activation through JumpReLU to handle inhibition.
- Expected residual attribution (ERA)Prior coactivation-based proxy for effectiveness.
- Fisher effectivenessSecond-order KL estimate of a weight's effect on outputs using Fisher metric.
- HelpfulnessAverage change in loss when a weight is ablated.
- Helpfulness mass criterionPreserving a fraction of sum of positive helpfulness to estimate density.
- Logit LensUnsupervised interpretability technique that projects activations through unembedding matrix; provides comparison point for NLA approach.
- Magnitude pruningBaseline pruning by virtual weight magnitude.
- Optimal Brain Damage / Optimal Brain SurgeonSecond-order saliency pruning; ancestor of Fisher effectiveness.
- Region of practical equivalence (ROPE) analysisEffect-size classification of sampled weights.
- Single-layer transcoderUsed to express MLP in monosemantic features, 4096 features with JumpReLU.
- Target-weighted ERA (TWERA)Target-weighted variant of ERA introduced in prior work.
Frameworks (8)
- Attribution-based Parameter Decomposition (APD)Decomposes parameters into faithful, minimal, simple components.
- Local learning coefficient (LLC)Singular learning theory measure of loss landscape degeneracy.
- Local Loss Landscape DecompositionSparsely active parameter-space subnetworks reconstructing loss gradients.
- Lottery ticket hypothesisSurviving subnetworks trainable in isolation.
- Singular learning theoryRelates parameters to loss via landscape geometry.
- Sparsely connected cross-layer transcodersMask virtual weights during training targeting interference.
- Stochastic Parameter Decomposition (SPD)Linear parameter decomposition with stochastic ablation.
- Virtual weight model (VW model)Rewriting of the 1L transformer into six families of virtual weights over tokens, positions, features, logits.
Datasets (2)
- Common Corpus (PleIAs)English and code training text for the transformer.
- Pleias-1.2B tokenizer (truncated)4096-token BPE vocabulary.
Findings (18)
- Newline feature 1254: Tokens→Features and OV position inputs anti-correlated (OLS slope -0.993, r²=0.912), summing to roughly constant bias
271 stratified contexts.
- Most effective weights are helpful, exceeding any harmful weight by an order of magnitude within each family (1,111-weight sample)
Three regimes by effectiveness.
- On ACETYLCHOLINE, direct, attention, MLP paths each favor wrong tokens (␣Mrs, ely, ATION) while 'E' is scored highly by all
No single path ranks E top.
- French word-end feature 3013: harmful weights remain intermingled with helpful across high-effectiveness range
Effectiveness doesn't fully clean.
- Fisher effectiveness is roughly lognormal across 10 orders of magnitude; median is 10,000× below max
Versus VW approx normal, median one third of max.
- Helpful and harmful weights are scattered across the entire range of virtual weight magnitudes
IN Tokens→Logits shows weak VW-helpfulness relationship.
- Fisher effectiveness is quantitatively similar to ERA/TWERA-style expected attribution
~27,000 point comparison.
- Transformer training loss fell from 8.93 to ~3.33 over 11,933 steps; test loss 3.38
Training results.
- VW model contains more helpful weights than parameters in original transformer
Even restricted to most effective and helpful.
- At 1% density the pruned VW model's performance is significantly compromised
Parameter count roughly matches original transformer.
Claims (12)
- Newline feature relies on attention path for activation signal and the Tokens→Features weight negates it for non-newline tokens; effectiveness highlights inhibition
Circuit interpretation.
- The large harmful IN→utions weight implies weight superposition since it would have no reason to be large if virtual weights were trained directly
Motivating example.
- A loss-minimizing model points its most consequential weights in good directions and has little reason to police weights barely reaching output
Interpretation of effectiveness tail result.
- Failure to reach sparsity reflects the basis, not the potential for sparse interpretation; no saliency scheme will do much better
Better bases may give sparsity.
- This note is the first demonstration of an interference weight inside a trained transformer by measuring effect on training loss
Novelty claim.
- Ineffective or harmful weights are interference weights the model carries alongside its actual circuits, not lessons learned
Interpretation of motivating example.
- Harmful weights reflect the tradeoff between implementing more or more complex circuits and the interference between them
Discussion.
- Pruning literature's finding of over-provisioned networks is consistent with the abundance of ineffective weights
Related work.
- Further sharpening of the effectiveness metric is not high-leverage compared to finding better bases
Helpfulness is direct measure and Fisher already prunes similarly.
- Large virtual weights are not inherently misleading but not guaranteed to be helpful or effective
Caution for practitioners.
Hypotheses (2)
- A better basis (e.g., factoring tokens into languages or parts of speech) may yield a sparse interpretable model
Same model can be dense in one basis and sparse in another.
- If all of ~2% retained weights were interpretable, the result would be closer to an 'upstairs' lift
Speculative possibility.
Questions (3)
- Do we assume that the model knows something we don't, or are these interference weights contributing noise?
Motivating question.
- Which of these signals implement functional circuits?
Motivating example.
- How does this feature know when to fire?
Newline feature puzzle.
Original abstract (expand)
This paper characterizes interference weights in a one-layer transformer by decomposing it into virtual weights between tokens, positions, features, and logits. We score weights along two axes: effectiveness (magnitude of effect on outputs) and helpfulness (magnitude and sign of effect on loss). We find that helpful and harmful weights are scattered across all weight magnitudes, that 70% of the least effective weights can be pruned with minimal loss impact, and that even the most effective and helpful weights still exceed the number of parameters in the original transformer, suggesting room for further sparsity improvements in better bases.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Weight-sparse transformers have interpretable circuitsAchyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, Dan Mossing Leo Gao2025≈ 83%
- ≈ 82%
- Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the TransformerLorenzo Noci, Mikhail Khodak, Mufan Li Yihe Dong2025≈ 81%
- Probing the Impact of Scale on Data-Efficient, Generalist Transformer World Models for AtariJooyeon Kim2026≈ 80%
- ≈ 80%
- Learning on Model Weights using Tree ExpertsBar Cavia, Jonathan Kahana, Yedid Hoshen Eliahu Horwitz2025≈ 80%
- Diversity of Transformer Layers: One Aspect of Parameter Scaling LawsYing Zhang, Jingun Kwon, Katsuhiko Hayashi, Manabu Okumura, Taro Watanabe Hidetaka Kamigaito2025≈ 80%
- Quantifying LLM Attention-Head Stability: Implications for Circuit UniversalityJack Stanley, Praneet Suresh, Danilo Bzdok Karan Bali2026≈ 80%
- ≈ 80%
- Interpreting Language Model Parametersin corpus2026≈ 80%
- Weight Patching: Toward Source-Level Mechanistic Localization in LLMsChengsheng Zhang, Guanzheng Qin, Rui Dai, Xinmei Tian Chenghao Sun2026≈ 80%
- Monet: Mixture of Monosemantic Experts for TransformersYoung Jin Ahn, Kee-Eung Kim, Jaewoo Kang Jungwoo Park2025≈ 80%
- The MASK Benchmark: Disentangling Honesty From Accuracy in AI SystemsArunim Agarwal, Mantas Mazeika, Cristina Menghini, Robert Vacareanu, Brad Kenstler, Mick Yang, Isabelle Barrass, Alice Gatti, Xuwang Yin, Eduardo Trevino, Matias Geralnik, Adam Khoja, Dean Lee, Summer Yue, Dan Hendrycks Richard Ren2026≈ 79%
- A Mathematical Framework for Transformer Circuitsin corpus2021≈ 79%
- ≈ 79%
- Visual Representations inside the Language ModelAmita Kamath, Madeleine Grunde-McLaughlin, Winson Han, Ranjay Krishna Benlin Liu2025≈ 79%
- ≈ 79%
- ≈ 79%
- ≈ 79%
- ≈ 79%
- ≈ 79%
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationsin corpus2026≈ 78%
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentionin corpus2026≈ 78%
- Evaluating Language Model Character Traitsin corpus2024≈ 78%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 78%
- Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationin corpus2026≈ 78%
- ≈ 78%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 78%
- ≈ 78%
- ≈ 78%