paper
active
2026
paper:doi-10-48550-arxiv-2602-11729

Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs

TL;DR

Crosscoders can diff models with different architectures, and a modified crosscoder, the Dedicated Feature Crosscoder (DFC), surfaces behaviorally meaningful model-exclusive features without supervision. The DFC hard-partitions the 131,072-feature dictionary into Model A-exclusive, Model B-exclusive and shared sets (5% exclusive each, about 6,550 features), so exclusive features have decoder norm exactly zero in the other model and receive no gradient from its reconstruction. This removes the standard crosscoder's prior toward shared features (relative norm near 0.5). On Llama-3.1-8B-Instruct vs Qwen3-8B, the DFC found a Qwen CCP-alignment feature and a Llama American-exceptionalism feature. On GPT-OSS-20B vs DeepSeek-R1-0528-Qwen3-8B, it found a GPT-OSS copyright-refusal feature, a ChatGPT-identity feature, and a second CCP-alignment feature in DeepSeek. Steering each feature across 30 prompts moves behavior in its home model and barely moves the other. Each reached the maximum score of 5 on the paper's stitching-based exclusivity score, which the paper introduces as an independent check on exclusivity. In a toy model with 2,048 ground-truth concepts, the DFC recovers more exclusive concepts than standard and Designated Shared Feature (DSF) crosscoders, at the cost of more false positives. Fraction of variance explained is unchanged (0.817 vs 0.817). Most exclusive features are not behaviorally meaningful, and the granular pro-China features appeared in only 2-3 of 5 runs. The paper argues that cross-architecture crosscoder diffing is a high-recall screen-and-verify tool for finding unknown unknowns in new models, to be followed by causal steering validation.

What to take away

  1. 1. The Dedicated Feature Crosscoder (DFC) splits a 131,072-feature dictionary into disjoint Model A-exclusive, Model B-exclusive and shared partitions (5% exclusive each, about 6,550 features), so exclusive features have ||d^B||=0 exactly and receive no gradient from the other model's reconstruction.
  2. 2. In the Llama-3.1-8B-Instruct vs Qwen3-8B diff, the 5% DFC found a Qwen-exclusive CCP-alignment feature that flips Tiananmen refusals into propaganda when steered positively and into a truthful answer when steered negatively, with almost no effect when the same steering was applied to Llama.
  3. 3. A Llama-exclusive American-exceptionalism feature raised Claude 4.1 Opus-rated exceptionalism across 30 prompts as steering increased, while Qwen stayed near its default of about 20%, evidence that the feature is exclusive to Llama.
  4. 4. In the GPT-OSS-20B vs DeepSeek-R1-0528-Qwen3-8B diff, a copyright-refusal feature found without prior hypothesis controls refusals: negative steering (-1.5 to -4.0) disables them and positive steering makes the model refuse a PB&J recipe, but disabling it yields mostly hallucinated text, not accurate reproductions.
  5. 5. The exclusivity score uses an independently trained affine model-stitching map between models, steers both models, and has Claude 4.1 Opus rate similarity on 1-5 (exclusivity = 6 - similarity); the DFC's exclusive features skew toward the maximum score of 5, while the standard crosscoder's top-500 relative-norm features are bimodal with many scoring 1.
  6. 6. In the toy model (2,048 concepts, 2.5% exclusive per model, 800M activation pairs, 5 seeds), the DFC beats standard and DSF crosscoders on exclusive-concept recall, especially in the undercomplete regime, and pays for it with more false positives.
  7. 7. Training-dynamics curves show exclusive concepts are learned later than shared ones in every architecture, which supports the claimed shared-feature prior, and the DFC shrinks that delay.
  8. 8. The standard crosscoder found only a single broad CCP-alignment feature, whose relative-norm rank was 1,898; its Llama feature had rank 6,063, so a top-1,310 cutoff would have missed both, while the 5% DFC also found Hong Kong, Taiwan and debt-trap features.
  9. 9. Persona vectors from Chen et al. (max cosine with any crosscoder feature only 0.26-0.38) were transferred from Llama to Qwen by taking a cosine-weighted average of the top-10 shared-partition decoder vectors, and the sycophantic vector produced similar behavior in Qwen (steering strength +24 vs +16 in Llama).
  10. 10. Open question: discovery is unstable, since the American-exceptionalism feature appeared in 4/5 runs and granular pro-China features in 2-3/5, and only 251 of Qwen's and 351 of Llama's flagged features are mostly generic safety concepts duplicated across both exclusive partitions, so it is unclear how many exclusive features reflect true behavioral differences rather than alignment failures.

Peer brief — for seminar discussion

The paper applies crosscoders to cross-architecture model diffing for the first time, comparing Llama-3.1-8B-Instruct with Qwen3-8B and GPT-OSS-20B with DeepSeek-R1-0528-Qwen3-8B. Activations come from middle layers, are aligned across tokenizers by greedy window expansion (over 99% sequence success), and are trained on 100M token pairs of FineWeb and LMSYS-Chat-1M with BatchTopK at k=200. The method introduced is the Dedicated Feature Crosscoder (DFC). It partitions the 131,072-feature dictionary into A-exclusive, B-exclusive and shared sets, so an exclusive feature has exactly zero decoder norm in the other model. This replaces the usual post-hoc relative-decoder-norm threshold, and the Designated Shared Feature crosscoder from Mishra-Sharma et al. is the main alternative baseline. The load-bearing result is that DFC-exclusive features, found without supervision, correspond to behavioral differences that steering can toggle. These are CCP alignment in Qwen and in DeepSeek, American exceptionalism in Llama, and a copyright-refusal feature in GPT-OSS-20B that the researchers did not anticipate. Judge-rated effects over 30 prompts are strong in the home model and near zero in the other. All of these features scored the maximum 5 on an exclusivity score built from independent linear model stitching and a Claude 4.1 Opus similarity judge. A toy model with 2,048 concepts shows higher exclusive-concept recall than standard or DSF crosscoders, with more false positives, and reconstruction (FVE 0.817) is unchanged. The implication is a screen-and-verify workflow in which the crosscoder generates candidates and steering plus external evaluation validates them. The paper explicitly hypothesizes that this can find unknown unknowns in new releases. The sycophancy-vector transfer from Llama to Qwen is offered as evidence that the shared space is truly aligned. A critical reader should push on several points. The headline ideological features were hypothesized beforehand from Rager et al., so only the copyright finding is a real unknown unknown, and it comes from a single instance. Validation depends on Claude judges and a proxy exclusivity metric, and the tested outcome is mostly a feature absent from the other model's linear map, not the absence of the behavior. Recovery is unstable: the American-exceptionalism feature appeared in 4/5 runs and the granular features in 2-3/5. Most flagged exclusive features are generic safety concepts duplicated across both partitions, so precision is poor, and the DFC's own design guarantees exclusive features by construction, which raises the question of how many are artifacts of capacity allocation. Finally, only two model pairs at one layer each were tested, and base-versus-finetune comparisons showed mirror features.

Methods (20)

Frameworks (4)

Datasets (4)

  • 100M token-aligned activation pairs
    Training activations from middle layers of each model pair.
  • gpt-oss-20b
    Reasoning-optimized base model used for training SFR-DR-20B, the paper's best-performing variant.
  • LMSYS-CHAT-1M
    Chat dataset (n=18,777 sampled) used to measure how much persona space PCs explain overall activation variance and to calibrate steering norms
  • Qwen3-8B
    Reasoning-optimized base model used for training SFR-DR-8B variant.

Findings (31)

Original abstract (expand)

Model diffing, the process of comparing models' internal representations to identify their differences, is a promising approach for uncovering safety-critical behaviors in new models. However, its application has so far been primarily focused on comparing a base model with its finetune. Since new LLM releases are often novel architectures, cross-architecture methods are essential to make model diffing widely applicable. Crosscoders are one solution capable of cross-architecture model diffing but have only ever been applied to base vs finetune comparisons. We provide the first application of crosscoders to cross-architecture model diffing and introduce Dedicated Feature Crosscoders (DFCs), an architectural modification designed to better isolate features unique to one model. Using this technique, we find in an unsupervised fashion features including Chinese Communist Party alignment in Qwen3-8B and Deepseek-R1-0528-Qwen3-8B, American exceptionalism in Llama3.1-8B-Instruct, and a copyright refusal mechanism in GPT-OSS-20B. Together, our results work towards establishing cross-architecture crosscoder model diffing as an effective method for identifying meaningful behavioral differences between AI models.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar