thinker:thomas-icardThomas Icard
Authored papers (2)
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse Minkowski sum of low-dimensional manifolds. The paper introduces block-sparse featurizers (BSFs) in three variants: Vanilla (top-k blocks by norm, untied encoder), Grassmannian (tied encoder with orthonormal charts), and Group Lasso (block soft-threshold). On a synthetic manifold-superposition toy, all three reach per-block R² of 0.93–0.97 against an oracle of 0.99, while a TopK SAE collapses to 0.53. On DINOv3 ViT-B, a minimum-description-length analysis at the 20% distortion floor shows blocks of dimension 2–4 describe activations in fewer bits than b=1 SAEs at every dictionary width and sparsity, and stable rank saturates near 3 even when b=16 is allowed. In InceptionV1, the curve-detector neurons and SAE atoms are fragments of one curve manifold, whose first three Fourier harmonics carry 89% of variance (59/18/12%) and survive a matched null by factors of 470 and 314. A DINOv3 block tracks sun luminance and shadow across twelve Blender models, and SDXL blocks support manifold steering, as in the pretzel block. The implication is that a featurizer is a hypothesis about the data-generating process, so interpretability should infer what the atoms must be rather than assume them. Blocks are one answer, matched to vision but probably not to language.
- Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations2023ⓒ 9
Distributed alignment search (DAS) resolves two blocking limitations of prior causal abstraction work—brute-force alignment search and the localist assumption that high-level variables map to disjoint neuron sets—by using gradient descent over orthogonal rotation matrices to find alignments in non-standard bases of neural representations. On a hierarchical equality task, a three-layer feed-forward network with hidden size 16 achieves 100% interchange intervention accuracy (IIA) under DAS at layer 1 with an 8-dimensional intervention subspace, whereas the best brute-force localist search reaches only 0.60 IIA and the closest localist alignment only 0.73 IIA. On the Monotonicity NLI benchmark, BERT-base fine-tuned on MoNLI achieves 100% IIA at layer 9 when 256 non-standard basis dimensions of the [CLS] token encode lexical entailment and 256 others encode negation, while no localist alignment exceeds 0.51 IIA on the same task. A subsequent subspace decomposition reveals a structural asymmetry: the hierarchical equality representations of w=x and y=z cannot be decomposed into representations of individual input identities (subspace DAS IIA ≈ 0.50–0.51), whereas the apparent lexical-entailment representation in BERT decomposes almost perfectly (IIA ≈ 0.97–0.98) into two word-identity representations. DAS implies that previous negative or weak causal abstraction findings may have been artifacts of the localist assumption, and that neural networks can genuinely implement tree-structured symbolic algorithms—but that apparent relational representations may sometimes be data structures over entity identities rather than true relational encodings.
More papers — OpenAlex / S2
Affiliations (1)
- Stanford University(institute)
Co-authors (12)
- Atticus Geiger15 shared
- Christopher Potts9 shared
- Noah D. Goodman9 shared
- Zhengxuan Wu9 shared
- Dron Hazra6 shared
- Ekdeep Singh Lubana6 shared
- Jack Merullo6 shared
- Lee Sharkey6 shared
- Matthew Kowal6 shared
- Mozes Jacobs6 shared
- Thomas Fel6 shared
- Thomas Serre6 shared
Their work is cited by (8)
- CausalGym: Benchmarking causal interpretability methods on linguistic tasks3× refs
- Addressing divergent representations from causal interventions on neural networks3× refs
- pyvene: A Library for Understanding and Improving PyTorch Models via Interventions3× refs
- Model Alignment Search3× refs
- Stochastic Parameter Decomposition3× refs
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?3× refs
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention3× refs
- Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts3× refs
Recent mentions (1)
- papers-typedgeiger-2023-finding-alignments.md