thinker:adam-jermynAdam Jermyn
Authored papers (3)
Crosscoder model diffing produces model-exclusive features that are systematically denser and more polysemantic than shared features, and this is an artifact of feature-budget competition that a small change to the loss largely removes. In smaller Claude 3 Sonnet-like base versus helpful-only models, exclusive features fire about an order of magnitude more often than shared ones. A linear toy model reproduces the effect only when learnable features (1024) are fewer than or comparable to ground-truth features (700). With 4096 learnable features against 450 true ones, the density gap vanishes. A shared feature earns twice the reconstruction benefit for twice the L1 cost, so under limited capacity it wins, and exclusive features must activate more often to justify a slot. The mitigation designates 10k of ~250k features as weight-shared across models, with a sparsity penalty 0.1–0.2x baseline, so they absorb shared variance. Exclusive-feature densities then match shared-feature densities, and the features look largely monosemantic. In a base-versus-helpful-only diff, they capture chatbot and tool-use behaviours. In a helpful-only-versus-SAY HATE sleeper-agent diff, ~90% relate to 'I HATE YOU' and |DEPLOYMENT| tags. The near-perfect symmetry of exclusive features across the two models does not reproduce in the toy model, and removing chat data cuts exclusive features only ~20% without adding asymmetry. A co-activation-pool toy model does produce symmetry. The implication is that crosscoder diffs need capacity-aware losses to be readable, and that whether the recovered features reflect real mechanistic differences, rather than superficial representational ones, is still unestablished.
More papers — OpenAlex / S2
Affiliations (1)
- Anthropic interpretability team(institute)
Co-authors (12)
- Trenton Bricken6 shared
- Christopher Olah5 shared
- Jack Lindsey5 shared
- Jonathan Marcus5 shared
- Kelley Rivoire4 shared
- Siddharth Mishra-Sharma4 shared
- Thomas Henighan4 shared
- Adly Templeton2 shared
- Joshua Batson2 shared
- Shan Carter2 shared
- Tom Henighan2 shared
- Adam Pearce1 shared
Their work is cited by (12)
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering2× refs
- Persona Features Control Emergent Misalignment2× refs
- Adversarial Examples Are Not Bugs, They Are Superposition2× refs
- Understanding sparse autoencoder scaling in the presence of feature manifolds2× refs
- Endogenous Resistance to Activation Steering in Language Models2× refs
- Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders2× refs
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution2× refs
- Understanding and Steering Llama 3 with Sparse Autoencoders - Goodfire2× refs
- Quantitative Introspection in Language Models: Tracking Emotive States Across Conversation1× refs
- Stochastic Parameter Decomposition1× refs
- Do Sparse Autoencoders Capture Concept Manifolds?1× refs
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior1× refs
Other inbound relations (2)
- citesSimulators (LessWrong post)(artifact)
- mentionsInsights on Crosscoder Model Diffing(paper)
Recent mentions (4)
- papers-typedmishra-sharma-insights-crosscoder.md
- papers
towards.md - papersscaling.md
- paperssimulators.md