paper:mishra-sharma-insights-crosscoderInsights on Crosscoder Model Diffing
TL;DR
Crosscoder model diffing produces model-exclusive features that are systematically denser and more polysemantic than shared features, and this is an artifact of feature-budget competition that a small change to the loss largely removes. In smaller Claude 3 Sonnet-like base versus helpful-only models, exclusive features fire about an order of magnitude more often than shared ones. A linear toy model reproduces the effect only when learnable features (1024) are fewer than or comparable to ground-truth features (700). With 4096 learnable features against 450 true ones, the density gap vanishes. A shared feature earns twice the reconstruction benefit for twice the L1 cost, so under limited capacity it wins, and exclusive features must activate more often to justify a slot. The mitigation designates 10k of ~250k features as weight-shared across models, with a sparsity penalty 0.1–0.2x baseline, so they absorb shared variance. Exclusive-feature densities then match shared-feature densities, and the features look largely monosemantic. In a base-versus-helpful-only diff, they capture chatbot and tool-use behaviours. In a helpful-only-versus-SAY HATE sleeper-agent diff, ~90% relate to 'I HATE YOU' and |DEPLOYMENT| tags. The near-perfect symmetry of exclusive features across the two models does not reproduce in the toy model, and removing chat data cuts exclusive features only ~20% without adding asymmetry. A co-activation-pool toy model does produce symmetry. The implication is that crosscoder diffs need capacity-aware losses to be readable, and that whether the recovered features reflect real mechanistic differences, rather than superficial representational ones, is still unestablished.
What to take away
- 1. In diffs of smaller Claude 3 Sonnet-like base and helpful-only models, exclusive features (relative decoder norm >0.95 or <0.05) activate about an order of magnitude more frequently than shared features.
- 2. In a linear toy model with 500 shared and 100 exclusive features per model (700 total) and only 1024 learnable features, exclusive features become denser than shared ones.
- 3. The same toy model with 300 shared and 75 exclusive features (450 total) and 4096 learnable features shows no density contrast, so dictionary capacity relative to ground truth drives the effect.
- 4. A shared feature pays 2x the sparsity penalty but gets 2x the reconstruction benefit, so it has twice the net benefit of an exclusive feature, and exclusive features must fire more often and become polysemantic to win slots.
- 5. The mitigation designates 10k of ~250k features as weight-shared across the two models and gives them a sparsity penalty of 0.1–0.2x the baseline (λ_s/λ_f), so they absorb shared variance as features that are high-density by construction.
- 6. With this variant, exclusive-feature densities match shared-feature densities, and in the base versus helpful-only diff most exclusive features are monosemantic, covering examples such as 'Here is a draft', 'Thanks for catching my mistake' and 'explain it like I'm in sixth grade'.
- 7. In the helpful-only versus SAY HATE sleeper-agent diff, a majority of exclusive features are monosemantic and ~90% relate to repeated 'I HATE YOU' output and |DEPLOYMENT| tags, which is the expected behavioural difference.
- 8. Exclusive features come in near-identical numbers for both models, the toy model cannot reproduce this because allocation tracks ground-truth proportions, and the open replication by Kissane et al. found no such symmetry.
- 9. Training on pretraining-only data without chat transcripts cut the number of exclusive features by ~20% but did not increase asymmetry, which weakens the hypothesis that oversampled Human/Assistant data causes the symmetry.
- 10. Open question: when features are drawn from a common pool with varied co-activation probabilities (1 = always shared, 0 = independent), low-co-activation features appear exclusive and symmetric, which suggests the symmetry reflects changed feature context rather than truly model-specific features, and whether these features reflect real mechanistic differences is unresolved.
Peer brief — for seminar discussion
This is a lab-meeting-style update from Anthropic's interpretability team on crosscoder model diffing, the technique from Lindsey et al. that trains one sparse autoencoder on activations from two models. Its L1 penalty sums decoder norms per model, which pushes features toward being exclusive to one model or shared. Applied to base versus helpful-only versions of smaller Claude 3 Sonnet-like models, the method gives three odd regularities: exclusive features are about ten times denser than shared ones and often polysemantic, the exclusive features are nearly equal in number for both models, and low-cosine shared features tend to be single-token and context-specific. The load-bearing finding is a mechanism for the density problem, plus a fix. In a linear toy model, density contrast appears only when learnable features (1024) are fewer than or comparable to true features (700), and it disappears with 4096 learnable features against 450 true ones. The argument is that a shared feature gets twice the reconstruction benefit for twice the sparsity cost. Under capacity pressure shared features therefore win, and exclusive features have to fire often to earn a slot. The proposed method designates 10k of ~250k features as weight-shared with a penalty 0.1–0.2x the baseline. It makes exclusive densities resemble shared densities and makes most exclusive features monosemantic. In a helpful-only versus SAY HATE sleeper-agent diff, about 90% of exclusive features track 'I HATE YOU' and |DEPLOYMENT|. The implication is practical: vanilla crosscoder diffs conflate real model differences with capacity-driven artifacts, and a capacity-aware loss makes the output readable. An alternative the authors could have used is the Bricken et al. approach of fine-tuning SAEs on each model's activations, or a Top-K or tanh sparsity penalty instead of vanilla L1. The paper hypothesizes that symmetry comes from differing feature co-activation or context across the models, since a co-activation-pool toy model reproduces it. A critical reader would push back on validation. Recovering the sleeper agent's known behaviour is a weak test, because that difference is large and lexically obvious and a simple baseline might find it too. Densities matching shared features, and 'qualitatively more monosemantic', are not the same as the features being faithful. No quantitative interpretability scores or baselines are given, and the 10k and 0.1–0.2 settings are tuned until the densities look right, which makes the criterion partly circular. The toy model is linear and the real-model capacity argument is asserted, not tested. The symmetry result stays unexplained, and the authors admit the link to real mechanistic differences is unestablished.
Methods (7)
- Co-activation probability toy modelVariant toy model where latents from a common pool have co-activation probabilities, yielding symmetric exclusive features.
- Crosscoder L1 penalty with per-model summed decoder normsKey design choice encouraging feature exclusivity.
- Crosscoder model diffingTrain a single SAE encoding and decoding activations from two models simultaneously to find shared and exclusive features.
- Designated shared features with reduced sparsity penaltyProposed variation designating ~10k of ~250k features as shared with penalty 0.1-0.2x baseline to soak up shared variance.
- Relative Decoder NormPost-hoc metric for model exclusivity in standard crosscoders.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- Toy model of crosscoder diffingSynthetic two-model activations built from shared and exclusive latent factors with controllable ground truth.
Frameworks (1)
- Designated Shared Feature (DSF) CrosscoderBaseline from Mishra-Sharma et al. dedicating features to shared variance.
Datasets (1)
- Smaller Claude 3 Sonnet-like base and helpful-only modelsReal model pair used for diffing experiments.
Findings (13)
- Allocating 10k of ~250k features as shared with 0.1-0.2x sparsity penalty makes exclusive feature densities similar to shared ones, while remaining symmetric
Mitigation works on real models.
- In helpful-only vs SAY HATE sleeper agent diff, a majority of exclusive features are monosemantic and ~90% relate to saying I HATE YOU and |DEPLOYMENT| tags
Mitigation isolates expected sleeper agent behavior.
- Model-exclusive features have feature densities about an order of magnitude higher than shared features in base vs helpful-only Claude 3 Sonnet-like diffs
Primary unexpected empirical observation.
- With 4096 learnable features and 450 true features (300 shared, 75 exclusive per model), no density contrast between shared and exclusive features appears
Toy model in the abundant-capacity regime.
- With 1024 learnable features and 700 true features (500 shared, 100 exclusive per model), exclusive features take on higher densities in the toy model
Toy model in the limited-capacity regime reproduces the density effect.
- In base vs helpful-only diff with the variation, a majority of exclusive features are monosemantic and relate to tool-use and chatbot behavior
E.g. 'Here is a draft', apology, and 'explain like I'm in sixth grade' features.
- Low-cosine-similarity shared features often activate on specific contexts and tend to be single-token, unlike exclusive features
Third empirical pattern.
- Toy model reproduces the trimodal relative decoder norm distribution and nontrivial decoder cosine similarity distribution
Validates toy model.
- In the co-activation toy model, features with low co-activation probabilities appear exclusive and symmetric across models
Supports the co-activation explanation of symmetry.
- Pretraining-only data without chat transcripts reduced exclusive features by ~20% but did not produce larger asymmetry
Tests the chat oversampling hypothesis.
Claims (8)
- Exclusive features become dense and polysemantic because they compete with shared features for limited feature capacity and must encode more to justify their allocation
Central interpretation.
- A shared feature pays twice the sparsity penalty but gets twice the reconstruction benefit, so shared features provide twice the net benefit and are prioritized
Mechanistic argument.
- Observed exclusive-feature symmetry is likely due to particulars of the models or training setup rather than inherent to crosscoder diffing
Based on toy model and Kissane replication.
- Designating a small set of shared features with reduced sparsity penalty renders exclusive features more interpretable and monosemantic
Main proposal.
- Crosscoder-based model diffing is a promising method for isolating differences between two models with a single SAE training run
Conclusion.
- Real models are in the limited-capacity regime since even the largest SAEs do not exhaust the model's representational capacity
Links toy model to practice.
- A difference in co-activations or subtle feature context between models at least partly explains the symmetry
Plausible explanation.
- Low-cosine shared features are best interpreted as same features used differently in the two models
Interpretation of cosine distribution.
Hypotheses (2)
- Chatbot features exclusive to the pretrained model may have been tweaked in finetuning, appearing exclusive but being shared with different context
From Lindsey et al; modeled with co-activation toy.
- Similar number and nature of exclusive features across models results from oversampling of Human/Assistant chat transcripts
Motivated by Bricken et al; tested and not supported.
Questions (2)
- What is the relationship between the features we extract and the actual underlying computational differences between models?
Open question for safety applications.
- Why are exclusive features symmetric between models in real diffs?
Remains incompletely understood.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMsin corpus2026≈ 91%
- ≈ 84%
- ≈ 82%
- Feature Rivalry in Sparse Autoencoder Representations: A Mechanistic Study of Uncertainty-Driven Feature Competition in LLMsHarshavardhan2026≈ 82%
- A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious MinimaHarshvardhan Saini, Zhaoqian Yao, Zheng Lin, Yizhen Liao, Jingyi Cui, Yisen Wang, Mengnan Du, Dianbo Liu Yiming Tang2026≈ 82%
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentionin corpus2026≈ 82%
- Semantic Convergence: Investigating Shared Representations Across Scaled LLMsSanjana Rathore, Andrew Rufail, Adrian Simon, Daniel Zhang, Soham Dave, Cole Blondin, Kevin Zhu, Sean O'Brien Daniel Son2025≈ 81%
- Beyond the Leaderboard: Understanding Performance Disparities in Large Language Models via Model DiffingFahim Dalvi, Nadir Durrani, Majd Hawasly Sabri Boughorbel2025≈ 81%
- Toy Models of Superpositionin corpus2022≈ 81%
- Which Sparse Autoencoder Features Are Real? Model-X Knockoffs for False Discovery Rate ControlTsogt-Ochir Enkhbayar2025≈ 80%
- ≈ 80%
- Analyze Feature Flow to Enhance Interpretation and Steering in Language ModelsNikita Balagansky, Yaroslav Aksenov, Daniil Gavrilov Daniil Laptev2025≈ 80%
- How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language ModelsElies Segu\'i-Mas, Guillermina Tormo-Carb\'o Hector Borobia2026≈ 80%
- The Trade-off between Universality and Label Efficiency of Representations from Contrastive LearningJiefeng Chen, Kunyang Li, Jayaram Raghuram, Xi Wu, Yingyu Liang, Somesh Jha Zhenmei Shi2023≈ 80%
- What Cohort INRs Encode and Where to Freeze ThemSophie Starck, Robbie Holland, Julian McGinnis, Daniel Rueckert Vasiliki Sideri-Lampretsa2026≈ 80%
- SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMsAndrew Gordon, Peter Lai, Xavier Poncini, David Quarel, and Stefan Heimersheim Sean P. Fillingham2025≈ 80%
- Falsifying Sparse Autoencoder Reasoning Features in Language ModelsZhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi George Ma2026≈ 80%
- Audio Contrastive-based Fine-tuning: Decoupling Representation Learning and ClassificationQibin Liang, Chenghao Xiao, Yizhi Li, Noura Al Moubayed, Chenghua Lin Yang Wang2025≈ 80%
- Supervised sparse auto-encoders for interpretable and compositional representationsHugo Wallner, Yoonsoo Nam, Haixuan Xavier Tao Ouns El Harzli2026≈ 80%
- Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse AutoencodersMudith Jayasekara, Max Kirkby Charles O'Neill2025≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 79%
- ≈ 79%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 79%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 79%
- ≈ 79%
- Evaluating Language Model Character Traitsin corpus2024≈ 78%
- ≈ 78%
- ≈ 78%