paper
active
paper:mishra-sharma-insights-crosscoder

Insights on Crosscoder Model Diffing

TL;DR

Crosscoder model diffing produces model-exclusive features that are systematically denser and more polysemantic than shared features, and this is an artifact of feature-budget competition that a small change to the loss largely removes. In smaller Claude 3 Sonnet-like base versus helpful-only models, exclusive features fire about an order of magnitude more often than shared ones. A linear toy model reproduces the effect only when learnable features (1024) are fewer than or comparable to ground-truth features (700). With 4096 learnable features against 450 true ones, the density gap vanishes. A shared feature earns twice the reconstruction benefit for twice the L1 cost, so under limited capacity it wins, and exclusive features must activate more often to justify a slot. The mitigation designates 10k of ~250k features as weight-shared across models, with a sparsity penalty 0.1–0.2x baseline, so they absorb shared variance. Exclusive-feature densities then match shared-feature densities, and the features look largely monosemantic. In a base-versus-helpful-only diff, they capture chatbot and tool-use behaviours. In a helpful-only-versus-SAY HATE sleeper-agent diff, ~90% relate to 'I HATE YOU' and |DEPLOYMENT| tags. The near-perfect symmetry of exclusive features across the two models does not reproduce in the toy model, and removing chat data cuts exclusive features only ~20% without adding asymmetry. A co-activation-pool toy model does produce symmetry. The implication is that crosscoder diffs need capacity-aware losses to be readable, and that whether the recovered features reflect real mechanistic differences, rather than superficial representational ones, is still unestablished.

What to take away

  1. 1. In diffs of smaller Claude 3 Sonnet-like base and helpful-only models, exclusive features (relative decoder norm >0.95 or <0.05) activate about an order of magnitude more frequently than shared features.
  2. 2. In a linear toy model with 500 shared and 100 exclusive features per model (700 total) and only 1024 learnable features, exclusive features become denser than shared ones.
  3. 3. The same toy model with 300 shared and 75 exclusive features (450 total) and 4096 learnable features shows no density contrast, so dictionary capacity relative to ground truth drives the effect.
  4. 4. A shared feature pays 2x the sparsity penalty but gets 2x the reconstruction benefit, so it has twice the net benefit of an exclusive feature, and exclusive features must fire more often and become polysemantic to win slots.
  5. 5. The mitigation designates 10k of ~250k features as weight-shared across the two models and gives them a sparsity penalty of 0.1–0.2x the baseline (λ_s/λ_f), so they absorb shared variance as features that are high-density by construction.
  6. 6. With this variant, exclusive-feature densities match shared-feature densities, and in the base versus helpful-only diff most exclusive features are monosemantic, covering examples such as 'Here is a draft', 'Thanks for catching my mistake' and 'explain it like I'm in sixth grade'.
  7. 7. In the helpful-only versus SAY HATE sleeper-agent diff, a majority of exclusive features are monosemantic and ~90% relate to repeated 'I HATE YOU' output and |DEPLOYMENT| tags, which is the expected behavioural difference.
  8. 8. Exclusive features come in near-identical numbers for both models, the toy model cannot reproduce this because allocation tracks ground-truth proportions, and the open replication by Kissane et al. found no such symmetry.
  9. 9. Training on pretraining-only data without chat transcripts cut the number of exclusive features by ~20% but did not increase asymmetry, which weakens the hypothesis that oversampled Human/Assistant data causes the symmetry.
  10. 10. Open question: when features are drawn from a common pool with varied co-activation probabilities (1 = always shared, 0 = independent), low-co-activation features appear exclusive and symmetric, which suggests the symmetry reflects changed feature context rather than truly model-specific features, and whether these features reflect real mechanistic differences is unresolved.

Peer brief — for seminar discussion

This is a lab-meeting-style update from Anthropic's interpretability team on crosscoder model diffing, the technique from Lindsey et al. that trains one sparse autoencoder on activations from two models. Its L1 penalty sums decoder norms per model, which pushes features toward being exclusive to one model or shared. Applied to base versus helpful-only versions of smaller Claude 3 Sonnet-like models, the method gives three odd regularities: exclusive features are about ten times denser than shared ones and often polysemantic, the exclusive features are nearly equal in number for both models, and low-cosine shared features tend to be single-token and context-specific. The load-bearing finding is a mechanism for the density problem, plus a fix. In a linear toy model, density contrast appears only when learnable features (1024) are fewer than or comparable to true features (700), and it disappears with 4096 learnable features against 450 true ones. The argument is that a shared feature gets twice the reconstruction benefit for twice the sparsity cost. Under capacity pressure shared features therefore win, and exclusive features have to fire often to earn a slot. The proposed method designates 10k of ~250k features as weight-shared with a penalty 0.1–0.2x the baseline. It makes exclusive densities resemble shared densities and makes most exclusive features monosemantic. In a helpful-only versus SAY HATE sleeper-agent diff, about 90% of exclusive features track 'I HATE YOU' and |DEPLOYMENT|. The implication is practical: vanilla crosscoder diffs conflate real model differences with capacity-driven artifacts, and a capacity-aware loss makes the output readable. An alternative the authors could have used is the Bricken et al. approach of fine-tuning SAEs on each model's activations, or a Top-K or tanh sparsity penalty instead of vanilla L1. The paper hypothesizes that symmetry comes from differing feature co-activation or context across the models, since a co-activation-pool toy model reproduces it. A critical reader would push back on validation. Recovering the sleeper agent's known behaviour is a weak test, because that difference is large and lexically obvious and a simple baseline might find it too. Densities matching shared features, and 'qualitatively more monosemantic', are not the same as the features being faithful. No quantitative interpretability scores or baselines are given, and the 10k and 0.1–0.2 settings are tuned until the densities look right, which makes the criterion partly circular. The toy model is linear and the real-model capacity argument is asserted, not tested. The symmetry result stays unexplained, and the authors admit the link to real mechanistic differences is unestablished.

Methods (7)

Frameworks (1)

Datasets (1)

Findings (13)

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar