thinker:nicholas-l-turnerNicholas L. Turner
Authored papers (2)
Interference weights are measurable inside a trained transformer, and the model keeps its most effective virtual weights helpful while confining harmful ones to a low-effectiveness range. In a one-layer, 2.9M-parameter model (d_m=256, 4,096-token vocab) expanded into a 331M-weight virtual weight (VW) model of tokens, positions, transcoder features and logits, Fisher effectiveness (a second-order KL estimate in nats) paired with helpfulness (the loss change under ablation) separates function from noise. The largest IN→"utions" Tokens→Logits weight never fires correctly in training, and its effectiveness sits roughly three orders of magnitude below the best weights from the same token. Raw virtual weight magnitude is a weak guide: effectiveness varies over about 10 orders of magnitude, and the median weight is 10,000× below the maximum. Pruning the least effective 70% costs about 0.01 nats and 85% costs under 0.1, yet the VW model stays dense. Helpfulness sampling finds 47.6% of weights with positive mean helpfulness, and preserving 99% of helpfulness mass needs 13.6% density. That is still tens of millions of weights, more than the original parameter count. Effectiveness is necessary but not sufficient for reading global circuits. Removing interference in the token and feature basis will not yield a sparse model. Progress depends on finding a better basis, with these metrics serving as the test for recognizing one.
More papers — OpenAlex / S2
Affiliations (1)
- Anthropic(institute)
Co-authors (12)
- Joshua Batson5 shared
- Jeffrey Wu4 shared
- Adam Jermyn1 shared
- Adam Pearce1 shared
- Adly Templeton1 shared
- Andy Jones1 shared
- Brian Chen1 shared
- C. Daniel Freeman1 shared
- Callum McDougall1 shared
- Chris Olah1 shared
- Craig Citro1 shared
- Edward Rees1 shared
Their work is cited by (8)
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering1× refs
- Persona Features Control Emergent Misalignment1× refs
- Adversarial Examples Are Not Bugs, They Are Superposition1× refs
- Understanding sparse autoencoder scaling in the presence of feature manifolds1× refs
- Endogenous Resistance to Activation Steering in Language Models1× refs
- Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders1× refs
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution1× refs
- Understanding and Steering Llama 3 with Sparse Autoencoders - Goodfire1× refs
Other inbound relations (1)
- mentionsCharacterizing interference weights in a tiny language model(paper)
Recent mentions (2)
- papers-typedcharacterizing-interference-067ab3.md
- papersscaling.md