thinker
active
thinker:ben-thompson

Ben Thompson

Author, Anthropic.

Authored
1
Introduces
0
Studies
0
Affiliations
1
Cited by
0

Authored papers (1)

  • Training a sparse autoencoder on the mean residual-stream activation of a conversation turn, rather than on individual tokens, yields features that describe a turn's high-level properties more completely than per-token features do, and makes attribution graphs tractable at long context. The instrument is the turn-averaged SAE (BatchTopK, d_sae=32,768, k=128) trained on Qwen-2.5-7B-Instruct with LMSYS-Chat-1M, plus a nested Matryoshka-style variant that splits 16,384 turn-level and 16,384 per-token features. Coverage and discrimination rank oppositely: per-token max-pooled features win 10-way matching at 95.0% against 73.9% for turn-averaged, but turn-averaged features win 87.9% of Sonnet 4.6 pairwise coverage comparisons, take mean rank 2.03 in 5-way ranking, and lead a Qwen3-Embedding-8B similarity metric (0.663 vs 0.617 for per-token max). Turn-level attribution graphs cut a hypothetical 10-turn, 250-token conversation from ~128K to ~5K nodes. In a 14-turn conversation that erodes from refusal to degeneration, they surface a jailbreak-detection feature at L16 that per-token aggregation never finds. Nested features reach the highest completeness and replacement scores despite slightly worse per-token nMSE (0.167 vs 0.162). Averaging acts as a low-pass filter on the token dimension, so reshaping the training signal can steer SAE features toward concept-level granularity, with implications for turn-wide tasks such as persona and safety monitoring over growing contexts.

More papers — OpenAlex / S2

Affiliations (1)

Co-authors (4)

Recent mentions (1)