thinker:lucius-bushnaqLucius Bushnaq
Authored papers (5)
VPD (adVersarial Parameter Decomposition) decomposes weight matrices directly into rank-one interpretable subcomponents rather than decomposing activations as sparse autoencoders (SAEs) do, flipping the standard mechanistic interpretability substrate. Applied to a 67M-parameter, 4-layer transformer trained on The Pile, VPD produces subcomponents that are sparse and interpretable while avoiding the feature-splitting artifacts characteristic of SAE-based approaches, and achieves a better sparsity-reconstruction tradeoff than transcoders — a competing parameter-level baseline. The adversarial ablation procedure that enforces mechanistic faithfulness is the method's core technical innovation: subcomponents must survive targeted ablation pressure, ensuring they correspond to genuine computational roles rather than statistical artifacts. Attention computations are shown to distribute across heads through parameter subcomponents with distinct, legible functional roles. Because VPD operates on weights rather than activations, it also enables manual model editing through direct parameter manipulation, a capability not available to activation-patching or SAE-steering pipelines. The paper argues this implies that the SAE program's foundational choice — treat activations as the unit of analysis — is not merely one option among many but an unnecessary constraint, and that parameter-level decomposition may be the more faithful path to understanding what a model has learned, with the frontier-scale generalizability of VPD left as the central open question.
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse Minkowski sum of low-dimensional manifolds. The paper introduces block-sparse featurizers (BSFs) in three variants: Vanilla (top-k blocks by norm, untied encoder), Grassmannian (tied encoder with orthonormal charts), and Group Lasso (block soft-threshold). On a synthetic manifold-superposition toy, all three reach per-block R² of 0.93–0.97 against an oracle of 0.99, while a TopK SAE collapses to 0.53. On DINOv3 ViT-B, a minimum-description-length analysis at the 20% distortion floor shows blocks of dimension 2–4 describe activations in fewer bits than b=1 SAEs at every dictionary width and sparsity, and stable rank saturates near 3 even when b=16 is allowed. In InceptionV1, the curve-detector neurons and SAE atoms are fragments of one curve manifold, whose first three Fourier harmonics carry 89% of variance (59/18/12%) and survive a matched null by factors of 470 and 314. A DINOv3 block tracks sun luminance and shadow across twelve Blender models, and SDXL blocks support manifold steering, as in the pretzel block. The implication is that a featurizer is a hypothesis about the data-generating process, so interpretability should infer what the atoms must be rather than assume them. Blocks are one answer, matched to vision but probably not to language.
Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.
Rank-one subcomponents trained against stochastic ablation masks recover ground-truth mechanisms in toy networks that Attribution-based Parameter Decomposition (APD) could not decompose, while removing APD's top-k hyperparameter, gradient attributions and shrinkage. Stochastic Parameter Decomposition (SPD) splits each weight matrix into C rank-one subcomponents that sum to the original weights. A tiny per-subcomponent MLP with a hard-sigmoid output predicts causal importance g in [0,1]. Each subcomponent is then masked by a random scalar drawn from U(g,1), and an importance-minimality loss pushes g toward zero. With S=1 mask sample, SPD reaches MMCS of 1.000 and mean L2 ratio of 0.992 to 1.031 on TMS5-2 and TMS40-10, with and without a hidden identity matrix. APD's ML2R was about 0.9. SPD recovers the m2+m1 expected subcomponents in TMS+ID, a case where APD fails. In the 100-feature, 50-neuron compressed-computation residual MLP it finds 100 W_in subcomponents, one per feature, plus a single rank-50 W_out component. This contradicts APD's earlier claim of 100 W_out components. It also decomposes 2-layer (100 features) and 3-layer (102 features, 51 neurons) cross-layer models, where the rank-50 and rank-51 components span all layers. These results suggest that sampling ablations and learning to predict importance is a more stable route than top-k with attributions. The authors argue this removes the main barriers to scaling linear parameter decomposition toward language models, though subcomponent clustering and scale remain unsolved.
More papers — OpenAlex / S2
Originates (1)
Affiliations (2)
- Apollo Research(institute)
- Goodfire(institute)
Co-authors (12)
- Lee Sharkey22 shared
- Atticus Geiger12 shared
- Dan Braun10 shared
- Dron Hazra6 shared
- Ekdeep Singh Lubana6 shared
- Jack Merullo6 shared
- Matthew Kowal6 shared
- Max Tegmark6 shared
- Mozes Jacobs6 shared
- Neel Nanda6 shared
- Thomas Fel6 shared
- Thomas Serre6 shared
Their work is cited by (7)
- From Memorization to Reasoning in the Spectrum of Loss Curvature3× refs
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?2× refs
- Model Alignment Search2× refs
- Do Sparse Autoencoders Capture Concept Manifolds?2× refs
- Tracing Persona Vectors Through LLM Pretraining2× refs
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds2× refs
- Steering at the Source: Style Modulation Heads for Robust Persona Control2× refs
Other inbound relations (2)
- mentionsFrom Memorization to Reasoning in the Spectrum of Loss Curvature(paper)
- mentionsStochastic Parameter Decomposition(paper)
Recent mentions (3)
- papers-typedbushnaq-2025-stochastic-parameter.md
- papers-typedjack-2025-memorization-reasoning.md
- papers-typedbushnaq-goodfire-vpd-parameters-2026.md