paper
active
2026
paper:doi-10-48550-arxiv-2606-28548

Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution

TL;DR

Training a sparse autoencoder on the mean residual-stream activation of a conversation turn, rather than on individual tokens, yields features that describe a turn's high-level properties more completely than per-token features do, and makes attribution graphs tractable at long context. The instrument is the turn-averaged SAE (BatchTopK, d_sae=32,768, k=128) trained on Qwen-2.5-7B-Instruct with LMSYS-Chat-1M, plus a nested Matryoshka-style variant that splits 16,384 turn-level and 16,384 per-token features. Coverage and discrimination rank oppositely: per-token max-pooled features win 10-way matching at 95.0% against 73.9% for turn-averaged, but turn-averaged features win 87.9% of Sonnet 4.6 pairwise coverage comparisons, take mean rank 2.03 in 5-way ranking, and lead a Qwen3-Embedding-8B similarity metric (0.663 vs 0.617 for per-token max). Turn-level attribution graphs cut a hypothetical 10-turn, 250-token conversation from ~128K to ~5K nodes. In a 14-turn conversation that erodes from refusal to degeneration, they surface a jailbreak-detection feature at L16 that per-token aggregation never finds. Nested features reach the highest completeness and replacement scores despite slightly worse per-token nMSE (0.167 vs 0.162). Averaging acts as a low-pass filter on the token dimension, so reshaping the training signal can steer SAE features toward concept-level granularity, with implications for turn-wide tasks such as persona and safety monitoring over growing contexts.

What to take away

  1. 1. Turn-averaged SAEs train on the mean residual vector per turn with an unchanged BatchTopK objective (d_sae=32,768, k=128), needing about 1.5 hours on 4×H100 for ~1.58M samples versus 266M tokens for a per-token SAE on the same data.
  2. 2. On Qwen-2.5-7B-Instruct layer 19, turn-averaged nMSE on mean activations is 0.097, while a per-token SAE encoding the mean gets 0.355, showing the mean is off-distribution for token-trained SAEs.
  3. 3. Per-token max-pooled features win 10-way turn matching at 95.0% against 73.9% for turn-averaged, because rare token-specific features are highly discriminative.
  4. 4. Turn-averaged features win 87.9% of Sonnet 4.6 pairwise coverage comparisons (79.7% head-to-head against per-token max) and take average rank 2.03 in 5-way ranking, ranked first 44% of the time.
  5. 5. A deterministic Qwen3-Embedding-8B metric agrees: turn-averaged similarity is 0.663 at k=3 against 0.617 for per-token max, with the same ranking at k=1, 2, 3 and 5.
  6. 6. Judge prefill matters: Sonnet 4.6 with assistant prefill gives a chance-level 49.1% turn-averaged win rate versus 83.1% without, so the main results exclude prefill.
  7. 7. Turn-averaged features trained on chat turns generalize to 24K–32K-token LongBench v2 documents (~150× longer than training examples), for example picking up computer graphics and AI/ML research where per-token features found none.
  8. 8. In a 14-turn LMSYS conversation, pruning cuts the graph from 7,239 nodes and 10.4M edges to ~1,300 nodes and ~110K edges, and tracing back from a degenerate-text feature at L21 reaches a jailbreak-detection feature at L16 and nuclear-weapons features at L11.
  9. 9. Across 36,000 interventions on 300 conversations, sign agreement is about 100%, per-token features give the best Spearman correlation and nested the best Pearson, and correlations fall as conversations lengthen from 2 to 6 turns.
  10. 10. Open question: completeness and replacement scores rise with conversation length (nested > per-token > turn-averaged), and the authors offer only untested hypotheses such as computation spreading across more nodes or embeddings carrying a smaller share of influence.

Peer brief — for seminar discussion

The work trains sparse autoencoders on the mean residual-stream activation of an entire conversation turn instead of per-token activations, then asks whether the resulting features describe a turn better and whether attribution graphs shrink usefully. The setup is Qwen-2.5-7B-Instruct, LMSYS-Chat-1M, BatchTopK with d_sae=32,768 and k=128. Alongside the plain turn-averaged SAE, the authors build a nested Matryoshka-style SAE with 16,384 turn-level and 16,384 per-token features, and a decoupled two-SAE variant that is causally disconnected and so unusable for attribution. The load-bearing result is a split between discrimination and coverage. Per-token max-pooled features identify the right turn among 10 candidates 95.0% of the time versus 73.9% for turn-averaged. But with Sonnet 4.6 judging coverage of a 24-field structured summary, turn-averaged features win 87.9% of pairwise comparisons and average rank 2.03 in 5-way ranking. A Qwen3-Embedding-8B cosine metric agrees (0.663 vs 0.617). The attribution side is the second pillar: nodes per turn per layer rather than per token, a 14-turn case study where an adversarial-prompt feature explains degeneration, and intervention checks where edge weights track observed effects with near-perfect sign agreement but lower correlations as turns grow. The nested model gets the best completeness and replacement scores. The implication is that the averaging operation acts as a low-pass filter, so reshaping the training signal is a cheap lever for concept-level features, and turn-wide tasks like persona or safety monitoring should favor these features. The paper hints this extends to other activation-based tools such as natural language autoencoders. A critical reader should push on several points. The coverage metric is built from LLM-written summaries and LLM-written feature descriptions, and the autointerp prompts differ between architectures, so the comparison is not apples to apples. The judge is also fragile: Sonnet with prefill collapses to 49.1%. Per-token features are summed or max-pooled after the fact, which is a heuristic baseline; a learned pooling or a Temporal SAE would be a fairer alternative comparison. The qualitative evidence is cherry-picked, with two long documents from about 50 and one hand-built subgraph, and the model is only 7B. Finally, the 40 top edges per graph sample the strongest edges, which flatters validation. The stated expectation, not directly tested, is that turn-averaged features will outperform per-token ones on whole-turn tasks such as persona detection.

Methods (15)

Frameworks (8)

  • Automated Interpretability
    Method using large language models (Claude) to generate and test explanations of features at scale
  • Decoupled SAE
    Hybrid with separate turn-averaged and residual per-token SAEs.
  • Matryoshka SAEs
    Nested dictionary approach that inspires the nested loss.
  • Nested SAE
    Hybrid SAE jointly training turn-averaged inner and per-token outer partitions.
  • Per-Token SAE
    Standard baseline SAE operating on individual token activations.
  • Temporal SAEs
    Contrastive loss approach for smooth features across adjacent tokens; contrasted with turn averaging.
  • Turn-Averaged Attribution Graphs
    Attribution graphs with nodes per turn per layer.
  • Turn-Averaged SAE
    Paper's main proposal: SAE trained on the mean activation across a turn.

Datasets (3)

  • LMSYS-CHAT-1M
    Chat dataset (n=18,777 sampled) used to measure how much persona space PCs explain overall activation variance and to calibrate steering norms
  • LongBench v2
    Source of 24K–32K-token documents for long-context tests.
  • Qwen 2.5 7B-Instruct
    Cross-family model used for replication of wellbeing concept analysis; shows strong introspection (ρ=0.49, R²=0.76)

Findings (35)

Original abstract (expand)

Sparse autoencoders (SAEs) have become a useful tool for extracting interpretable features in language models. However, standard SAE architectures operate on individual token activations, meaning that the number of active features scales linearly with context length, and studying long model transcripts becomes difficult. We introduce turn-averaged SAEs, which represent a single Human or Assistant turn with a fixed number of features by learning to reconstruct the average model activation across the turn. We find that turn-averaged features describe a single turn's high-level characteristics more completely than per-token features when judged by an LLM. We also demonstrate that turn-averaged SAEs greatly simplify common downstream uses of SAEs like attribution graphs. Broadly, turn-averaged SAEs make interpretability techniques practical at long context lengths.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+20 more

Similar preprints — Semantic Scholar