thinker
active
thinker:bart-bussmann

Bart Bussmann

Co-author of VPD paper; contributes to Goodfire mechanistic interpretability efforts.

Authored
3
Introduces
1
Studies
1
Affiliations
1
Cited by
5

Authored papers (3)

  • VPD (adVersarial Parameter Decomposition) decomposes weight matrices directly into rank-one interpretable subcomponents rather than decomposing activations as sparse autoencoders (SAEs) do, flipping the standard mechanistic interpretability substrate. Applied to a 67M-parameter, 4-layer transformer trained on The Pile, VPD produces subcomponents that are sparse and interpretable while avoiding the feature-splitting artifacts characteristic of SAE-based approaches, and achieves a better sparsity-reconstruction tradeoff than transcoders — a competing parameter-level baseline. The adversarial ablation procedure that enforces mechanistic faithfulness is the method's core technical innovation: subcomponents must survive targeted ablation pressure, ensuring they correspond to genuine computational roles rather than statistical artifacts. Attention computations are shown to distribute across heads through parameter subcomponents with distinct, legible functional roles. Because VPD operates on weights rather than activations, it also enables manual model editing through direct parameter manipulation, a capability not available to activation-patching or SAE-steering pipelines. The paper argues this implies that the SAE program's foundational choice — treat activations as the unit of analysis — is not merely one option among many but an unnecessary constraint, and that parameter-level decomposition may be the more faithful path to understanding what a model has learned, with the frontier-scale generalizability of VPD left as the central open question.

  • Training a sparse autoencoder on the mean residual-stream activation of a conversation turn, rather than on individual tokens, yields features that describe a turn's high-level properties more completely than per-token features do, and makes attribution graphs tractable at long context. The instrument is the turn-averaged SAE (BatchTopK, d_sae=32,768, k=128) trained on Qwen-2.5-7B-Instruct with LMSYS-Chat-1M, plus a nested Matryoshka-style variant that splits 16,384 turn-level and 16,384 per-token features. Coverage and discrimination rank oppositely: per-token max-pooled features win 10-way matching at 95.0% against 73.9% for turn-averaged, but turn-averaged features win 87.9% of Sonnet 4.6 pairwise coverage comparisons, take mean rank 2.03 in 5-way ranking, and lead a Qwen3-Embedding-8B similarity metric (0.663 vs 0.617 for per-token max). Turn-level attribution graphs cut a hypothetical 10-turn, 250-token conversation from ~128K to ~5K nodes. In a 14-turn conversation that erodes from refusal to degeneration, they surface a jailbreak-detection feature at L16 that per-token aggregation never finds. Nested features reach the highest completeness and replacement scores despite slightly worse per-token nMSE (0.167 vs 0.162). Averaging acts as a low-pass filter on the token dimension, so reshaping the training signal can steer SAE features toward concept-level granularity, with implications for turn-wide tasks such as persona and safety monitoring over growing contexts.

More papers — OpenAlex / S2

Affiliations (1)

Co-authors (11)