paper:doi-10-48550-arxiv-2606-28548Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution
TL;DR
Training a sparse autoencoder on the mean residual-stream activation of a conversation turn, rather than on individual tokens, yields features that describe a turn's high-level properties more completely than per-token features do, and makes attribution graphs tractable at long context. The instrument is the turn-averaged SAE (BatchTopK, d_sae=32,768, k=128) trained on Qwen-2.5-7B-Instruct with LMSYS-Chat-1M, plus a nested Matryoshka-style variant that splits 16,384 turn-level and 16,384 per-token features. Coverage and discrimination rank oppositely: per-token max-pooled features win 10-way matching at 95.0% against 73.9% for turn-averaged, but turn-averaged features win 87.9% of Sonnet 4.6 pairwise coverage comparisons, take mean rank 2.03 in 5-way ranking, and lead a Qwen3-Embedding-8B similarity metric (0.663 vs 0.617 for per-token max). Turn-level attribution graphs cut a hypothetical 10-turn, 250-token conversation from ~128K to ~5K nodes. In a 14-turn conversation that erodes from refusal to degeneration, they surface a jailbreak-detection feature at L16 that per-token aggregation never finds. Nested features reach the highest completeness and replacement scores despite slightly worse per-token nMSE (0.167 vs 0.162). Averaging acts as a low-pass filter on the token dimension, so reshaping the training signal can steer SAE features toward concept-level granularity, with implications for turn-wide tasks such as persona and safety monitoring over growing contexts.
What to take away
- 1. Turn-averaged SAEs train on the mean residual vector per turn with an unchanged BatchTopK objective (d_sae=32,768, k=128), needing about 1.5 hours on 4×H100 for ~1.58M samples versus 266M tokens for a per-token SAE on the same data.
- 2. On Qwen-2.5-7B-Instruct layer 19, turn-averaged nMSE on mean activations is 0.097, while a per-token SAE encoding the mean gets 0.355, showing the mean is off-distribution for token-trained SAEs.
- 3. Per-token max-pooled features win 10-way turn matching at 95.0% against 73.9% for turn-averaged, because rare token-specific features are highly discriminative.
- 4. Turn-averaged features win 87.9% of Sonnet 4.6 pairwise coverage comparisons (79.7% head-to-head against per-token max) and take average rank 2.03 in 5-way ranking, ranked first 44% of the time.
- 5. A deterministic Qwen3-Embedding-8B metric agrees: turn-averaged similarity is 0.663 at k=3 against 0.617 for per-token max, with the same ranking at k=1, 2, 3 and 5.
- 6. Judge prefill matters: Sonnet 4.6 with assistant prefill gives a chance-level 49.1% turn-averaged win rate versus 83.1% without, so the main results exclude prefill.
- 7. Turn-averaged features trained on chat turns generalize to 24K–32K-token LongBench v2 documents (~150× longer than training examples), for example picking up computer graphics and AI/ML research where per-token features found none.
- 8. In a 14-turn LMSYS conversation, pruning cuts the graph from 7,239 nodes and 10.4M edges to ~1,300 nodes and ~110K edges, and tracing back from a degenerate-text feature at L21 reaches a jailbreak-detection feature at L16 and nuclear-weapons features at L11.
- 9. Across 36,000 interventions on 300 conversations, sign agreement is about 100%, per-token features give the best Spearman correlation and nested the best Pearson, and correlations fall as conversations lengthen from 2 to 6 turns.
- 10. Open question: completeness and replacement scores rise with conversation length (nested > per-token > turn-averaged), and the authors offer only untested hypotheses such as computation spreading across more nodes or embeddings carrying a smaller share of influence.
Peer brief — for seminar discussion
The work trains sparse autoencoders on the mean residual-stream activation of an entire conversation turn instead of per-token activations, then asks whether the resulting features describe a turn better and whether attribution graphs shrink usefully. The setup is Qwen-2.5-7B-Instruct, LMSYS-Chat-1M, BatchTopK with d_sae=32,768 and k=128. Alongside the plain turn-averaged SAE, the authors build a nested Matryoshka-style SAE with 16,384 turn-level and 16,384 per-token features, and a decoupled two-SAE variant that is causally disconnected and so unusable for attribution. The load-bearing result is a split between discrimination and coverage. Per-token max-pooled features identify the right turn among 10 candidates 95.0% of the time versus 73.9% for turn-averaged. But with Sonnet 4.6 judging coverage of a 24-field structured summary, turn-averaged features win 87.9% of pairwise comparisons and average rank 2.03 in 5-way ranking. A Qwen3-Embedding-8B cosine metric agrees (0.663 vs 0.617). The attribution side is the second pillar: nodes per turn per layer rather than per token, a 14-turn case study where an adversarial-prompt feature explains degeneration, and intervention checks where edge weights track observed effects with near-perfect sign agreement but lower correlations as turns grow. The nested model gets the best completeness and replacement scores. The implication is that the averaging operation acts as a low-pass filter, so reshaping the training signal is a cheap lever for concept-level features, and turn-wide tasks like persona or safety monitoring should favor these features. The paper hints this extends to other activation-based tools such as natural language autoencoders. A critical reader should push on several points. The coverage metric is built from LLM-written summaries and LLM-written feature descriptions, and the autointerp prompts differ between architectures, so the comparison is not apples to apples. The judge is also fragile: Sonnet with prefill collapses to 49.1%. Per-token features are summed or max-pooled after the fact, which is a heuristic baseline; a learned pooling or a Temporal SAE would be a fairer alternative comparison. The qualitative evidence is cherry-picked, with two long documents from about 50 and one hand-built subgraph, and the model is only 7B. Finally, the 40 top edges per graph sample the strongest edges, which flatters validation. The stated expectation, not directly tested, is that turn-averaged features will outperform per-token ones on whole-turn tasks such as persona detection.
Methods (15)
- 10-way matching evaluationDiscrimination evaluation with 9 distractors.
- 24-field structured summary schemaClaude-generated 24 fields in 5 categories used as reference for coverage.
- 5-way ranking evaluationCoverage evaluation ranking all five feature lists.
- Attribution GraphsGradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.
- Completeness and replacement metricsMetrics of how thoroughly features explain computation.
- Contrastive-prompt persona feature pipelineRank features by activation ratio on positive vs negative rollouts.
- Embedding coverage metricDeterministic cosine-similarity coverage measure.
- Intervention validationPerturbing source features along decoder direction and measuring target change.
- Max pooling of per-token featuresAlternative aggregation giving high discrimination.
- Normalized mean squared error (nMSE)Reconstruction quality metric.
- Pairwise ranking evaluationCoverage evaluation comparing two feature lists.
- Per-paragraph attribution case studyAttribution share by system prompt paragraph relative to neutral baseline.
- Persona VectorsDirections in activation space encoding a model's dispositional self-presentation, used as evidence for self-modelling.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- Sum pooling of per-token featuresAggregation of token-level activations.
Frameworks (8)
- Automated InterpretabilityMethod using large language models (Claude) to generate and test explanations of features at scale
- Decoupled SAEHybrid with separate turn-averaged and residual per-token SAEs.
- Matryoshka SAEsNested dictionary approach that inspires the nested loss.
- Nested SAEHybrid SAE jointly training turn-averaged inner and per-token outer partitions.
- Per-Token SAEStandard baseline SAE operating on individual token activations.
- Temporal SAEsContrastive loss approach for smooth features across adjacent tokens; contrasted with turn averaging.
- Turn-Averaged Attribution GraphsAttribution graphs with nodes per turn per layer.
- Turn-Averaged SAEPaper's main proposal: SAE trained on the mean activation across a turn.
Datasets (3)
- LMSYS-CHAT-1MChat dataset (n=18,777 sampled) used to measure how much persona space PCs explain overall activation variance and to calibrate steering norms
- LongBench v2Source of 24K–32K-token documents for long-context tests.
- Qwen 2.5 7B-InstructCross-family model used for replication of wellbeing concept analysis; shows strong introspection (ρ=0.49, R²=0.76)
Findings (35)
- Turn-averaged features recover outline structure, automotive domain, and regulatory compliance in the blog outline; per-token finds only outline and lexical features
Example C.
- Turn-averaged and per-token SAEs both surface persona-relevant features (evil, hallucination, sycophancy, poetic, etc.) via contrastive rollouts, with noise
Appendix I.
- Turn-averaged attribution subgraph traces degenerate output to jailbreak-detection feature at L16 and nuclear weapons/AI safety features in earlier turns
14-turn conversation case study.
- On a 30,023-token computer graphics paper, turn-averaged features identify six aspects while per-token misses graphics and AI/ML research
LongBench v2 example E.
- Turn-averaged features flag an incorrect answer (digit-exclusion puzzle) while per-token features find none of the error recognition
Example A.
- Per-token features have strongest Spearman correlation and nested the strongest Pearson; turn-averaged weaker but significant
Figure 6.
- Sonnet without prefill (83.1%), Haiku with prefill (82.7%) and Haiku without prefill (62.7%) all favor turn-averaged features
Judge sensitivity.
- Per-token graph lacks any adversarial prompt detection feature; attribution flows to 'chemical compound names' and 'JSON'
Contrast in §6.3.
- Attribution to the target system paragraph rises above neutral baseline for all 8 queries across all three architectures
Table 2; e.g., Identity +1.8 for turn-averaged.
- Nested features achieve highest completeness and replacement, ordering nested > per-token > turn-averaged across lengths
Figure 7.
Claims (11)
- Training SAEs on mean hidden state yields features capturing high-level turn characteristics (topic, style, behavior) rather than lexical patterns
Central claim.
- Discrimination and coverage give opposite rankings: per-token max is easiest to identify, turn-averaging covers the full range
§5.5.
- Reshaping the training signal can compel SAE features toward desired characteristics and extends to other activation analyses
Conclusion.
- Nested architecture is better suited to turn-level attribution than per-token features aggregated after the fact
Based on sufficiency scores.
- Turn-averaged SAEs make interpretability practical at long context by decoupling feature count from token count
Abstract.
- Per-token SAE objective does not compel high-level characteristics to separate from token-level detail
Motivation.
- Turn-level attribution graphs give compact, interpretable explanations of multi-turn model behavior
Case study conclusion.
- Averaging acts as a low-pass filter that smooths away per-token detail and retains broad concepts
Intuition.
- Turn-averaged features generalize across span lengths, even ~150x beyond training length
Long-context results.
- Fully decoupled SAEs are causally disconnected and cannot support attribution
Per-token SAE input is invariant to uniform shifts from ablating turn-averaged features.
Hypotheses (3)
- Turn-averaged features will outperform per-token features on whole-turn tasks such as summarizing multi-turn exchanges or judging persona/safety traits
Prediction from coverage results.
- Longer conversations distribute computation across more feature nodes, or embeddings account for less influence, explaining rising completeness
Left to future work.
- Information concentration at specific tokens interacts with turn-averaging
Future work direction.
Questions (4)
- Can we use per-turn attribution graphs to represent how a model produces a particular response in a compact and tractable way for a lengthy multi-turn conversation?
Motivates 14-turn case.
- can a subset of features optimally reconstruct the mean of the activations while the full set of features reconstructs the per-token activations?
Motivates nested SAE.
- can we show that such graphs suitably represent the computation done by the model, and demonstrate their usefulness?
Motivates case studies.
- do they reflect actual model behavior under intervention?
Edge weights first-order approximation.
Original abstract (expand)
Sparse autoencoders (SAEs) have become a useful tool for extracting interpretable features in language models. However, standard SAE architectures operate on individual token activations, meaning that the number of active features scales linearly with context length, and studying long model transcripts becomes difficult. We introduce turn-averaged SAEs, which represent a single Human or Assistant turn with a fixed number of features by learning to reconstruct the average model activation across the turn. We find that turn-averaged features describe a single turn's high-level characteristics more completely than per-token features when judged by an LLM. We also demonstrate that turn-averaged SAEs greatly simplify common downstream uses of SAEs like attribution graphs. Broadly, turn-averaged SAEs make interpretability techniques practical at long context lengths.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 84%
- ≈ 83%
- CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder FeaturesZekun Wu, Adriano Koshiyama Seonglae Cho2026≈ 84%
- ≈ 83%
- A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model SteeringJiaqing Xie2025≈ 83%
- Stable and Steerable Sparse Autoencoders with Weight RegularizationOliver M. Crook Piotr Jedryszek2026≈ 83%
- Descriptive Collision in Sparse Autoencoder Auto-Interpretability: When One Explanation Describes Many FeaturesJordan F. McCann2026≈ 83%
- Revisiting End-To-End Sparse Autoencoder Training: A Short Finetune Is All You NeedAdam Karvonen2025≈ 83%
- Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEsAashiq Muhamed, Yujia Zheng, Lingjing Kong, Zeyu Tang, Mona T. Diab, Virginia Smith, Kun Zhang Xiangchen Song2025≈ 82%
- SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMsAndrew Gordon, Peter Lai, Xavier Poncini, David Quarel, and Stefan Heimersheim Sean P. Fillingham2025≈ 82%
- Measuring and Guiding MonosemanticityFelix Friedrich, Manuel Brack, Stephan W\"aldchen, Bj\"orn Deiseroth, Patrick Schramowski, Kristian Kersting Ruben H\"arle2025≈ 82%
- From Where Words Come: Efficient Regularization of Code Tokenizers Through Source AttributionPavel Chizhov and Egor Bogomolov and Ivan P. Yamshchikov2026≈ 82%
- Sparse Autoencoder Decomposition of Clinical Sequence Model Representations: Feature Complexity, Task Specialisation, and Mortality PredictionFeng Dong, Andreas Karwath Chris Sainsbury2026≈ 81%
- Step-Level Sparse Autoencoder for Reasoning Process InterpretationJiayu Liu, Yuhang Lai, Hao Xu, Zhenya Huang, Ning Miao Xuan Yang2026≈ 81%
- The Rate-Distortion-Polysemanticity Tradeoff in SAEsFrancesco Locatello Tommaso Mencattini and Francesco Montagna2026≈ 81%
- From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder FeaturesPablo Magari\~nos-Docampo, Javier Perez-Robles Ruben Fernandez-Boullon2026≈ 81%
- Pairwise matrices for sparse autoencoders: single-feature inspection mislabels causal axesMichael A. Riegler and Birk Sebastian Frostelid Torpmann-Hagen2026≈ 81%
- Which Sparse Autoencoder Features Are Real? Model-X Knockoffs for False Discovery Rate ControlTsogt-Ochir Enkhbayar2025≈ 81%
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationsin corpus2026≈ 80%
- ≈ 80%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 80%
- Facet-Level Persona Control by Trait-Activated Routing with Contrastive SAE for Role-Playing LLMsin corpus2026≈ 80%
- Persona Features Control Emergent Misalignmentin corpus2025≈ 80%
- ≈ 80%
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilityin corpus2025≈ 80%
- ≈ 80%
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalin corpus2026≈ 80%
- ≈ 80%
- ≈ 80%
- ≈ 79%
+20 more