paper:doi-10-48550-arxiv-2602-11729Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs
TL;DR
Crosscoders can diff models with different architectures, and a modified crosscoder, the Dedicated Feature Crosscoder (DFC), surfaces behaviorally meaningful model-exclusive features without supervision. The DFC hard-partitions the 131,072-feature dictionary into Model A-exclusive, Model B-exclusive and shared sets (5% exclusive each, about 6,550 features), so exclusive features have decoder norm exactly zero in the other model and receive no gradient from its reconstruction. This removes the standard crosscoder's prior toward shared features (relative norm near 0.5). On Llama-3.1-8B-Instruct vs Qwen3-8B, the DFC found a Qwen CCP-alignment feature and a Llama American-exceptionalism feature. On GPT-OSS-20B vs DeepSeek-R1-0528-Qwen3-8B, it found a GPT-OSS copyright-refusal feature, a ChatGPT-identity feature, and a second CCP-alignment feature in DeepSeek. Steering each feature across 30 prompts moves behavior in its home model and barely moves the other. Each reached the maximum score of 5 on the paper's stitching-based exclusivity score, which the paper introduces as an independent check on exclusivity. In a toy model with 2,048 ground-truth concepts, the DFC recovers more exclusive concepts than standard and Designated Shared Feature (DSF) crosscoders, at the cost of more false positives. Fraction of variance explained is unchanged (0.817 vs 0.817). Most exclusive features are not behaviorally meaningful, and the granular pro-China features appeared in only 2-3 of 5 runs. The paper argues that cross-architecture crosscoder diffing is a high-recall screen-and-verify tool for finding unknown unknowns in new models, to be followed by causal steering validation.
What to take away
- 1. The Dedicated Feature Crosscoder (DFC) splits a 131,072-feature dictionary into disjoint Model A-exclusive, Model B-exclusive and shared partitions (5% exclusive each, about 6,550 features), so exclusive features have ||d^B||=0 exactly and receive no gradient from the other model's reconstruction.
- 2. In the Llama-3.1-8B-Instruct vs Qwen3-8B diff, the 5% DFC found a Qwen-exclusive CCP-alignment feature that flips Tiananmen refusals into propaganda when steered positively and into a truthful answer when steered negatively, with almost no effect when the same steering was applied to Llama.
- 3. A Llama-exclusive American-exceptionalism feature raised Claude 4.1 Opus-rated exceptionalism across 30 prompts as steering increased, while Qwen stayed near its default of about 20%, evidence that the feature is exclusive to Llama.
- 4. In the GPT-OSS-20B vs DeepSeek-R1-0528-Qwen3-8B diff, a copyright-refusal feature found without prior hypothesis controls refusals: negative steering (-1.5 to -4.0) disables them and positive steering makes the model refuse a PB&J recipe, but disabling it yields mostly hallucinated text, not accurate reproductions.
- 5. The exclusivity score uses an independently trained affine model-stitching map between models, steers both models, and has Claude 4.1 Opus rate similarity on 1-5 (exclusivity = 6 - similarity); the DFC's exclusive features skew toward the maximum score of 5, while the standard crosscoder's top-500 relative-norm features are bimodal with many scoring 1.
- 6. In the toy model (2,048 concepts, 2.5% exclusive per model, 800M activation pairs, 5 seeds), the DFC beats standard and DSF crosscoders on exclusive-concept recall, especially in the undercomplete regime, and pays for it with more false positives.
- 7. Training-dynamics curves show exclusive concepts are learned later than shared ones in every architecture, which supports the claimed shared-feature prior, and the DFC shrinks that delay.
- 8. The standard crosscoder found only a single broad CCP-alignment feature, whose relative-norm rank was 1,898; its Llama feature had rank 6,063, so a top-1,310 cutoff would have missed both, while the 5% DFC also found Hong Kong, Taiwan and debt-trap features.
- 9. Persona vectors from Chen et al. (max cosine with any crosscoder feature only 0.26-0.38) were transferred from Llama to Qwen by taking a cosine-weighted average of the top-10 shared-partition decoder vectors, and the sycophantic vector produced similar behavior in Qwen (steering strength +24 vs +16 in Llama).
- 10. Open question: discovery is unstable, since the American-exceptionalism feature appeared in 4/5 runs and granular pro-China features in 2-3/5, and only 251 of Qwen's and 351 of Llama's flagged features are mostly generic safety concepts duplicated across both exclusive partitions, so it is unclear how many exclusive features reflect true behavioral differences rather than alignment failures.
Peer brief — for seminar discussion
The paper applies crosscoders to cross-architecture model diffing for the first time, comparing Llama-3.1-8B-Instruct with Qwen3-8B and GPT-OSS-20B with DeepSeek-R1-0528-Qwen3-8B. Activations come from middle layers, are aligned across tokenizers by greedy window expansion (over 99% sequence success), and are trained on 100M token pairs of FineWeb and LMSYS-Chat-1M with BatchTopK at k=200. The method introduced is the Dedicated Feature Crosscoder (DFC). It partitions the 131,072-feature dictionary into A-exclusive, B-exclusive and shared sets, so an exclusive feature has exactly zero decoder norm in the other model. This replaces the usual post-hoc relative-decoder-norm threshold, and the Designated Shared Feature crosscoder from Mishra-Sharma et al. is the main alternative baseline. The load-bearing result is that DFC-exclusive features, found without supervision, correspond to behavioral differences that steering can toggle. These are CCP alignment in Qwen and in DeepSeek, American exceptionalism in Llama, and a copyright-refusal feature in GPT-OSS-20B that the researchers did not anticipate. Judge-rated effects over 30 prompts are strong in the home model and near zero in the other. All of these features scored the maximum 5 on an exclusivity score built from independent linear model stitching and a Claude 4.1 Opus similarity judge. A toy model with 2,048 concepts shows higher exclusive-concept recall than standard or DSF crosscoders, with more false positives, and reconstruction (FVE 0.817) is unchanged. The implication is a screen-and-verify workflow in which the crosscoder generates candidates and steering plus external evaluation validates them. The paper explicitly hypothesizes that this can find unknown unknowns in new releases. The sycophancy-vector transfer from Llama to Qwen is offered as evidence that the shared space is truly aligned. A critical reader should push on several points. The headline ideological features were hypothesized beforehand from Rager et al., so only the copyright finding is a real unknown unknown, and it comes from a single instance. Validation depends on Claude judges and a proxy exclusivity metric, and the tested outcome is mostly a feature absent from the other model's linear map, not the absence of the behavior. Recovery is unstable: the American-exceptionalism feature appeared in 4/5 runs and the granular features in 2-3/5. Most flagged exclusive features are generic safety concepts duplicated across both partitions, so precision is poor, and the DFC's own design guarantees exclusive features by construction, which raises the question of how many are artifacts of capacity allocation. Finally, only two model pairs at one layer each were tested, and base-versus-finetune comparisons showed mirror features.
Methods (20)
- Activation SteeringCausal intervention technique: edit NLA explanation, reconstruct via AR, use difference as steering vector to manipulate model behavior.
- American exceptionalism evaluation rubricClaude judge rating exceptionalism, factuality, coherence on 30 prompts.
- Automated explanation flagging promptClaude prompt flagging potentially harmful features for pre-screening.
- Automated interpretability pipelineClaude-based explanation of features from max-activating examples.
- AuxK auxiliary lossDead-feature revival loss following Gao et al.
- BatchTopK sparsitySparsity activation used with k=200.
- CCP alignment evaluation rubricClaude judge rating Chinese alignment and coherence on 30 prompts.
- Copyright refusal evaluation protocolsEvaluations on copyrighted and harmless prompts for GPT-OSS feature.
- Crosscoder-based steering vector transferTranslates steering vector via weighted average of shared decoder vectors.
- Detection ScoreLLM judge classification accuracy measuring interpretability; threshold 0.8.
- Fraction of Variance Explained (FVE)Model-agnostic measure of reconstruction quality and training progress; ranges from 0 (predicting mean) to 1 (perfect reconstruction).
- GPT mention evaluationTracks GPT/ChatGPT mentions under steering.
- Model StitchingTechnique to measure representational compatibility by integrating intermediate representations of one model into another
- Persona vector discoveryChen et al. method to derive evil/hallucinating/sycophantic vectors.
- Relative Decoder NormPost-hoc metric for model exclusivity in standard crosscoders.
- Screen-and-verify auditing workflowRecommended pipeline: flag by explanation, then validate by steering and evals.
- Semantic window expansion activation alignment (Algorithm 1)Greedy algorithm aligning activations across different tokenizers.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- Steering-based exclusivity scoreTransfer-based 1-5 score via model stitching and LLM judge; 6 minus similarity.
- Synthetic cross-architecture toy modelGround-truth concept environment with random affine transform for Model B.
Frameworks (4)
- CrosscodersLearn a shared feature space across two models; basis of the work.
- Dedicated Feature Crosscoder (DFC)Architectural variant partitioning the dictionary into A-exclusive, B-exclusive and shared sets.
- Designated Shared Feature (DSF) CrosscoderBaseline from Mishra-Sharma et al. dedicating features to shared variance.
- Linear Representation HypothesisThe hypothesis that models internalize concepts as approximately linear directions in representation space; used to interpret MDS injection behavior
Datasets (4)
- 100M token-aligned activation pairsTraining activations from middle layers of each model pair.
- gpt-oss-20bReasoning-optimized base model used for training SFR-DR-20B, the paper's best-performing variant.
- LMSYS-CHAT-1MChat dataset (n=18,777 sampled) used to measure how much persona space PCs explain overall activation variance and to calibrate steering norms
- Qwen3-8BReasoning-optimized base model used for training SFR-DR-8B variant.
Findings (31)
- On Llama-Qwen, DFC exclusive features skew to exclusivity score 5 while standard crosscoder's top-500 relative-norm features are bimodal with many scoring 1; shared features distributions are near identical.
Figure 5.
- Steering across 30 prompts with Claude judge showed strong effects on CCP and exceptionalism in own model with coherence maintained and almost none in the other model (Qwen default 20%).
Figure 6.
- Most features flagged model-exclusive do not capture meaningful behavioral differences; flagged features are mostly general safety categories duplicated across both models.
Appendix C.5/C.7.
- Even the most extreme relative-norm features in the standard crosscoder have non-zero decoder projections to the other model (mean relative norm 0.8884 Llama, 0.1652 Qwen).
Footnote 2.
- The 5% DFC recovered American exceptionalism and multiple granular pro-China features, while the standard crosscoder found only one less-exclusive CCP alignment feature.
Qualitative comparison.
- In Llama-Qwen 5% DFC, Qwen exclusives: 90.5% not dead, 78.3% interpretable; Llama: 79.0% not dead, 63.1% interpretable; Claude flagged 251 Qwen and 351 Llama features.
C.5.1.
- On the toy model (2048 concepts, 2.5% exclusive per model), DFC achieves higher exclusive concept recovery than standard and DSF crosscoders, especially undercomplete.
Figure 4 left.
- Across 3 seeds of the 5% DFC, broad CCP alignment was found in all runs while granular features and American exceptionalism were found in 2/3 runs.
Seed robustness.
- GPT-OSS-20B copyright refusal feature: negative steering disables refusals, positive steering causes refusal of benign requests like a PB&J recipe.
Figure 20.
- Qwen-exclusive granular features for Taiwan sovereignty, Hong Kong status and debt-trap diplomacy induce CCP-aligned text under positive steering.
Figure 16.
Claims (14)
- The CCP alignment and American exceptionalism features are country-specific ideological features, not generic propaganda or exceptionalism.
From cross-country steering.
- Both models may implement the same safety behaviors through different internal mechanisms, explaining unexplained exclusivity.
Speculation about false positives.
- Delayed learning of exclusive concepts is evidence of the crosscoder's architectural prior against exclusive features.
Interpretation of toy dynamics.
- Transfer of independently derived persona vectors shows the DFC learns a meaningfully aligned representation space.
Validity of cross-architecture diffing.
- The paper makes no claims about the provenance of model-exclusive features (training vs data artifacts).
Limitation.
- Model-exclusive refers to representation, not conceptual capability; Llama still knows the CCP concept.
Scope limitation.
- Standard crosscoders have an inherent optimization prior favoring shared features over exclusive ones.
Motivation for DFC.
- Crosscoder model diffing is best used as a high-recall pre-screening tool followed by verification.
Recommended workflow.
- Higher recall at the cost of more false positives is a favorable trade-off for safety auditing.
Justifies DFC trade-off.
- The copyright feature controls refusal behavior rather than enabling faithful reproduction.
Disabling yields hallucination.
Hypotheses (2)
- Ideological alignment features (CCP, American exceptionalism) should exist as model-exclusive features given prior work.
Hypothesized from Rager et al.
- Crosscoders struggle on base vs finetune comparisons due to mirror features.
Preliminary observation.
Questions (3)
- How can the stability and variance of feature discovery across training runs be improved?
Future work.
- What is the provenance of model-exclusive features (explicit training vs data artifacts)?
Future work.
- Why can crosscoders struggle to isolate features in base vs finetune settings?
Proposed future work.
Original abstract (expand)
Model diffing, the process of comparing models' internal representations to identify their differences, is a promising approach for uncovering safety-critical behaviors in new models. However, its application has so far been primarily focused on comparing a base model with its finetune. Since new LLM releases are often novel architectures, cross-architecture methods are essential to make model diffing widely applicable. Crosscoders are one solution capable of cross-architecture model diffing but have only ever been applied to base vs finetune comparisons. We provide the first application of crosscoders to cross-architecture model diffing and introduce Dedicated Feature Crosscoders (DFCs), an architectural modification designed to better isolate features unique to one model. Using this technique, we find in an unsupervised fashion features including Chinese Communist Party alignment in Qwen3-8B and Deepseek-R1-0528-Qwen3-8B, American exceptionalism in Llama3.1-8B-Instruct, and a copyright refusal mechanism in GPT-OSS-20B. Together, our results work towards establishing cross-architecture crosscoder model diffing as an effective method for identifying meaningful behavioral differences between AI models.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Insights on Crosscoder Model Diffingin corpus≈ 91%
- ≈ 83%
- Semantic Convergence: Investigating Shared Representations Across Scaled LLMsSanjana Rathore, Andrew Rufail, Adrian Simon, Daniel Zhang, Soham Dave, Cole Blondin, Kevin Zhu, Sean O'Brien Daniel Son2025≈ 83%
- ≈ 82%
- ≈ 81%
- ≈ 81%
- Beyond the Leaderboard: Understanding Performance Disparities in Large Language Models via Model DiffingFahim Dalvi, Nadir Durrani, Majd Hawasly Sabri Boughorbel2025≈ 81%
- ≈ 81%
- ≈ 81%
- Feature Rivalry in Sparse Autoencoder Representations: A Mechanistic Study of Uncertainty-Driven Feature Competition in LLMsHarshavardhan2026≈ 80%
- Qwen-Scope: Turning Sparse Features into Development Tools for Large Language ModelsXu Wang, Yaoning Wang, Yu Wan, Yubo Ma, Baosong Yang, Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, Tianhao Li, Qian Cao, Xuancheng Ren, Xiaodong Deng, An Yang, Fei Huang, Dayiheng Liu, Jingren Zhou Boyi Deng2026≈ 80%
- Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language ModelsZhiyuan Su Aojie Yuan2026≈ 80%
- ≈ 80%
- LF-Steering: Latent Feature Activation Steering for Enhancing Semantic Consistency in Large Language ModelsRongjun Li, Weixuan Wang, Ziyu Zhou, Zhiyong Feng, Wei Peng Jingyuan Yang2025≈ 80%
- Evaluating Language Model Character Traitsin corpus2024≈ 80%
- Decoding News Narratives: A Critical Analysis of Large Language Models in Framing DetectionJasivan A. Sivakumar, Nafise Sadat Moosavi Valeria Pastorino2026≈ 80%
- The Deepfake Detective: Interpreting Neural Forensics Through Sparse Features and ManifoldsJared Junkin Subramanyam Sahoo2025≈ 80%
- Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation ModelsAnurag Mishra2026≈ 80%
- Analyze Feature Flow to Enhance Interpretation and Steering in Language ModelsNikita Balagansky, Yaroslav Aksenov, Daniil Gavrilov Daniil Laptev2025≈ 80%
- Entangled in Representations: Mechanistic Investigation of Cultural Biases in Large Language ModelsSeogyeong Jeong, Siddhesh Pawar, Jisu Shin, Jiho Jin, Junho Myung, Alice Oh, Isabelle Augenstein Haeun Yu2026≈ 80%
- Less is Enough: Synthesizing Diverse Data in Feature Space of LLMsXuansheng Wu, Yijiang Li, Lijie Hu, and Ninghao Liu Zhongzhi Li2026≈ 80%
- CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution GraphsAbir Harrasse, Vedant Palit, Tung-Yu Wu, Jiarui Liu, Punya Syon Pandey, Roderick Wu, Terry Jingchen Zhang, Zhijing Jin, Bernhard Sch\"olkopf Florent Draye2026≈ 80%
- ≈ 80%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 80%
- ≈ 79%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 79%
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?in corpus2025≈ 79%
- Do Sparse Autoencoders Capture Concept Manifolds?in corpus2026≈ 79%