thinker:lee-sharkeyLee Sharkey
Authored papers (6)
VPD (adVersarial Parameter Decomposition) decomposes weight matrices directly into rank-one interpretable subcomponents rather than decomposing activations as sparse autoencoders (SAEs) do, flipping the standard mechanistic interpretability substrate. Applied to a 67M-parameter, 4-layer transformer trained on The Pile, VPD produces subcomponents that are sparse and interpretable while avoiding the feature-splitting artifacts characteristic of SAE-based approaches, and achieves a better sparsity-reconstruction tradeoff than transcoders — a competing parameter-level baseline. The adversarial ablation procedure that enforces mechanistic faithfulness is the method's core technical innovation: subcomponents must survive targeted ablation pressure, ensuring they correspond to genuine computational roles rather than statistical artifacts. Attention computations are shown to distribute across heads through parameter subcomponents with distinct, legible functional roles. Because VPD operates on weights rather than activations, it also enables manual model editing through direct parameter manipulation, a capability not available to activation-patching or SAE-steering pipelines. The paper argues this implies that the SAE program's foundational choice — treat activations as the unit of analysis — is not merely one option among many but an unnecessary constraint, and that parameter-level decomposition may be the more faithful path to understanding what a model has learned, with the frontier-scale generalizability of VPD left as the central open question.
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse Minkowski sum of low-dimensional manifolds. The paper introduces block-sparse featurizers (BSFs) in three variants: Vanilla (top-k blocks by norm, untied encoder), Grassmannian (tied encoder with orthonormal charts), and Group Lasso (block soft-threshold). On a synthetic manifold-superposition toy, all three reach per-block R² of 0.93–0.97 against an oracle of 0.99, while a TopK SAE collapses to 0.53. On DINOv3 ViT-B, a minimum-description-length analysis at the 20% distortion floor shows blocks of dimension 2–4 describe activations in fewer bits than b=1 SAEs at every dictionary width and sparsity, and stable rank saturates near 3 even when b=16 is allowed. In InceptionV1, the curve-detector neurons and SAE atoms are fragments of one curve manifold, whose first three Fourier harmonics carry 89% of variance (59/18/12%) and survive a matched null by factors of 470 and 314. A DINOv3 block tracks sun luminance and shadow across twelve Blender models, and SDXL blocks support manifold steering, as in the pretzel block. The implication is that a featurizer is a hypothesis about the data-generating process, so interpretability should infer what the atoms must be rather than assume them. Blocks are one answer, matched to vision but probably not to language.
Mechanistic interpretability's dominant decomposition method, sparse dictionary learning (SDL), is not yet adequate for the field's engineering goals, and no method has yet been shown to beat non-interpretability baselines on real tasks. A 16-million-latent sparse autoencoder spliced into GPT-4 drops language-modeling loss to that of a model trained with roughly 10% of GPT-4's pretraining compute, and Makelov et al. report GPT-2 small performance falling 10% on task-specific data and 40% on the full distribution. The 27-author synthesis, led by Apollo Research and funded by Schmidt Sciences, organizes the field around a three-step reverse-engineering framework (decompose, describe, validate) set against concept-based interpretability, and uses circuit discovery as its case study of a full pipeline. Circuit faithfulness metrics depend on the intervention implementation, and circuits approximate average-case but not per-datapoint behavior. Most validated work sits on three families: CNNs, BERT and GPT. Reconstruction error, the linear-representation assumption, sparsity as a proxy for interpretability, unexplained feature geometry and dataset-dependent latents are all identified as unresolved. Applications such as white-box evaluations, mechanistic anomaly detection, steering and unlearning are tractable with shallow descriptions, but activation-based unlearning remains uncompetitive. The review argues that progress needs stronger theory of generalization, validation through model organisms and non-cherry-picked benchmarks, and head-to-head comparison against black-box baselines. It concludes that scientific and engineering goals should advance in parallel, and that overstated progress risks false assurance in governance.
Rank-one subcomponents trained against stochastic ablation masks recover ground-truth mechanisms in toy networks that Attribution-based Parameter Decomposition (APD) could not decompose, while removing APD's top-k hyperparameter, gradient attributions and shrinkage. Stochastic Parameter Decomposition (SPD) splits each weight matrix into C rank-one subcomponents that sum to the original weights. A tiny per-subcomponent MLP with a hard-sigmoid output predicts causal importance g in [0,1]. Each subcomponent is then masked by a random scalar drawn from U(g,1), and an importance-minimality loss pushes g toward zero. With S=1 mask sample, SPD reaches MMCS of 1.000 and mean L2 ratio of 0.992 to 1.031 on TMS5-2 and TMS40-10, with and without a hidden identity matrix. APD's ML2R was about 0.9. SPD recovers the m2+m1 expected subcomponents in TMS+ID, a case where APD fails. In the 100-feature, 50-neuron compressed-computation residual MLP it finds 100 W_in subcomponents, one per feature, plus a single rank-50 W_out component. This contradicts APD's earlier claim of 100 W_out components. It also decomposes 2-layer (100 features) and 3-layer (102 features, 51 neurons) cross-layer models, where the rank-50 and rank-51 components span all layers. These results suggest that sampling ablations and learning to predict importance is a more stable route than top-k with attributions. The authors argue this removes the main barriers to scaling linear parameter decomposition toward language models, though subcomponent clustering and scale remain unsolved.
A sparse autoencoder trained on layer-19 residual-stream activations of Llama-3-8B-Instruct, using LMSYS-Chat-1M, yields features that can be labeled automatically, ranked by causal relevance, and used to steer chat behavior, and training-data choice decides whether those features are usable for intervention. Goodfire built a ReLU SAE with MSE plus L1 loss, following the Anthropic April 2024 update settings. Its explicit rationale for avoiding TopK and JumpReLU was concern about high-frequency features. Claude labels each live feature from activation samples across the full activation range. Labels are scored contrastively against distractors and against a feature's ten nearest embedding neighbours. The attribution instrument is a gradient of the predicted-token logit minus the mean logit with respect to SAE features, summed over token positions. Interventions are scaled between −h_max and +h_max, and the SAE error term ε is held fixed. Chat-data SAEs found behaviourally relevant features, often on chat special tokens. Non-chat FineWeb data and diverse mixes gave more dead and low-frequency features and poorer intervention candidates. Goodfire conjectures that features become interpretable before they become intervenable. Negative interventions often need several features set below zero, which is consistent with echo features and self-repair. Some features resist steering, possibly because of cross-layer superposition. The implication is that SAE quality for steering depends on where activations were harvested and on causal validation. Interpretability alone does not establish it. Autointerp must also move beyond token-level labels.
More papers — OpenAlex / S2
Originates (1)
Studies (1)
Affiliations (2)
- Apollo Research(institute)
- Goodfire(institute)
Co-authors (12)
- Lucius Bushnaq22 shared
- Atticus Geiger18 shared
- Dan Braun10 shared
- Dron Hazra9 shared
- Ekdeep Singh Lubana9 shared
- Jack Merullo9 shared
- Matthew Kowal9 shared
- Max Tegmark9 shared
- Mozes Jacobs9 shared
- Neel Nanda9 shared
- Thomas Fel9 shared
- Thomas Serre9 shared
Their work is cited by (12)
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds4× refs
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?4× refs
- Steering at the Source: Style Modulation Heads for Robust Persona Control4× refs
- Do Sparse Autoencoders Capture Concept Manifolds?4× refs
- Tracing Persona Vectors Through LLM Pretraining3× refs
- Model Alignment Search3× refs
- From Memorization to Reasoning in the Spectrum of Loss Curvature3× refs
- Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers1× refs
- Endogenous Resistance to Activation Steering in Language Models1× refs
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior1× refs
- Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders1× refs
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution1× refs
Other inbound relations (4)
- mentionsOpen Problems in Mechanistic Interpretability(paper)
- mentionsStochastic Parameter Decomposition(paper)
- mentionsStructuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds(paper)
- mentionsUnderstanding and Steering Llama 3 with Sparse Autoencoders - Goodfire(paper)
Recent mentions (7)
- papers-typedunderstanding-steering-8b85ff.md
- papers-typedsharkey-2025-open-problems.md
- papers-typedbushnaq-2025-stochastic-parameter.md
- papers-typedfel-2026-structuring-sparsity.md
- papers
towards.md - paperssimulators.md
- papers-typedbushnaq-goodfire-vpd-parameters-2026.md