thinker:bart-bussmannBart Bussmann
Co-author of VPD paper; contributes to Goodfire mechanistic interpretability efforts.
Authored papers (3)
VPD (adVersarial Parameter Decomposition) decomposes weight matrices directly into rank-one interpretable subcomponents rather than decomposing activations as sparse autoencoders (SAEs) do, flipping the standard mechanistic interpretability substrate. Applied to a 67M-parameter, 4-layer transformer trained on The Pile, VPD produces subcomponents that are sparse and interpretable while avoiding the feature-splitting artifacts characteristic of SAE-based approaches, and achieves a better sparsity-reconstruction tradeoff than transcoders — a competing parameter-level baseline. The adversarial ablation procedure that enforces mechanistic faithfulness is the method's core technical innovation: subcomponents must survive targeted ablation pressure, ensuring they correspond to genuine computational roles rather than statistical artifacts. Attention computations are shown to distribute across heads through parameter subcomponents with distinct, legible functional roles. Because VPD operates on weights rather than activations, it also enables manual model editing through direct parameter manipulation, a capability not available to activation-patching or SAE-steering pipelines. The paper argues this implies that the SAE program's foundational choice — treat activations as the unit of analysis — is not merely one option among many but an unnecessary constraint, and that parameter-level decomposition may be the more faithful path to understanding what a model has learned, with the frontier-scale generalizability of VPD left as the central open question.
Training a sparse autoencoder on the mean residual-stream activation of a conversation turn, rather than on individual tokens, yields features that describe a turn's high-level properties more completely than per-token features do, and makes attribution graphs tractable at long context. The instrument is the turn-averaged SAE (BatchTopK, d_sae=32,768, k=128) trained on Qwen-2.5-7B-Instruct with LMSYS-Chat-1M, plus a nested Matryoshka-style variant that splits 16,384 turn-level and 16,384 per-token features. Coverage and discrimination rank oppositely: per-token max-pooled features win 10-way matching at 95.0% against 73.9% for turn-averaged, but turn-averaged features win 87.9% of Sonnet 4.6 pairwise coverage comparisons, take mean rank 2.03 in 5-way ranking, and lead a Qwen3-Embedding-8B similarity metric (0.663 vs 0.617 for per-token max). Turn-level attribution graphs cut a hypothetical 10-turn, 250-token conversation from ~128K to ~5K nodes. In a 14-turn conversation that erodes from refusal to degeneration, they surface a jailbreak-detection feature at L16 that per-token aggregation never finds. Nested features reach the highest completeness and replacement scores despite slightly worse per-token nMSE (0.167 vs 0.162). Averaging acts as a low-pass filter on the token dimension, so reshaping the training signal can steer SAE features toward concept-level granularity, with implications for turn-wide tasks such as persona and safety monitoring over growing contexts.
More papers — OpenAlex / S2
Originates (1)
Studies (1)
Affiliations (1)
- Goodfire(institute)
Co-authors (11)
- Harish Kamath3 shared
- Kevin Der3 shared
- Thompson, Ben2 shared
- Ben Thompson1 shared
- Dan Braun1 shared
- Lee Sharkey1 shared
- Linda Linsefors1 shared
- Lucius Bushnaq1 shared
- Michael Ivanitskiy1 shared
- Nathan Hu1 shared
- Oliver Clive-Griffin1 shared
Their work is cited by (5)
- Understanding sparse autoencoder scaling in the presence of feature manifolds1× refs
- Do Sparse Autoencoders Capture Concept Manifolds?1× refs
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior1× refs
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds1× refs
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution1× refs
Other inbound relations (2)
Recent mentions (3)
- papers-typedder-2026-turn-averaged.md
- papers-typedkarvonen-2025-saebench-comprehensive.md
- papers-typedbushnaq-goodfire-vpd-parameters-2026.md