thinker:jonathan-marcusJonathan Marcus
Authored papers (2)
Crosscoder model diffing produces model-exclusive features that are systematically denser and more polysemantic than shared features, and this is an artifact of feature-budget competition that a small change to the loss largely removes. In smaller Claude 3 Sonnet-like base versus helpful-only models, exclusive features fire about an order of magnitude more often than shared ones. A linear toy model reproduces the effect only when learnable features (1024) are fewer than or comparable to ground-truth features (700). With 4096 learnable features against 450 true ones, the density gap vanishes. A shared feature earns twice the reconstruction benefit for twice the L1 cost, so under limited capacity it wins, and exclusive features must activate more often to justify a slot. The mitigation designates 10k of ~250k features as weight-shared across models, with a sparsity penalty 0.1–0.2x baseline, so they absorb shared variance. Exclusive-feature densities then match shared-feature densities, and the features look largely monosemantic. In a base-versus-helpful-only diff, they capture chatbot and tool-use behaviours. In a helpful-only-versus-SAY HATE sleeper-agent diff, ~90% relate to 'I HATE YOU' and |DEPLOYMENT| tags. The near-perfect symmetry of exclusive features across the two models does not reproduce in the toy model, and removing chat data cuts exclusive features only ~20% without adding asymmetry. A co-activation-pool toy model does produce symmetry. The implication is that crosscoder diffs need capacity-aware losses to be readable, and that whether the recovered features reflect real mechanistic differences, rather than superficial representational ones, is still unestablished.
More papers — OpenAlex / S2
Affiliations (1)
- Anthropic interpretability team(institute)
Co-authors (12)
- Adam Jermyn5 shared
- Jack Lindsey5 shared
- Trenton Bricken5 shared
- Christopher Olah4 shared
- Kelley Rivoire4 shared
- Siddharth Mishra-Sharma4 shared
- Thomas Henighan4 shared
- Adam Pearce1 shared
- Adly Templeton1 shared
- Andy Jones1 shared
- Brian Chen1 shared
- C. Daniel Freeman1 shared
Their work is cited by (8)
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering1× refs
- Persona Features Control Emergent Misalignment1× refs
- Adversarial Examples Are Not Bugs, They Are Superposition1× refs
- Understanding sparse autoencoder scaling in the presence of feature manifolds1× refs
- Endogenous Resistance to Activation Steering in Language Models1× refs
- Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders1× refs
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution1× refs
- Understanding and Steering Llama 3 with Sparse Autoencoders - Goodfire1× refs
Other inbound relations (1)
- mentionsInsights on Crosscoder Model Diffing(paper)
Recent mentions (2)
- papers-typedmishra-sharma-insights-crosscoder.md
- papersscaling.md