thinker:satchel-grantSatchel Grant
Authored papers (4)
Resampling noise in reasoning-chain uncertainty dynamics is almost exactly multinomial, so most of the cost of Forking Paths Analysis re-measures structure that neighboring tokens already contain. On tinyMMLU (n=100) with Llama-3-8B-Instruct and DeepSeek-R1-Distill-Llama-8B, 1.77B tokens of S=200 reference data show log-TVD falling with log S at slopes of −0.4903 (development set), −0.4757 (Llama) and −0.4838 (DeepSeek), against the −1/2 expected under i.i.d. multinomial sampling. Measured replicate TVD sits within 0.98–1.01 of the exact multinomial null. The paper's estimator, a three-stage smoother, exploits this: PELT change-point detection with an exact multinomial cost segments the base path at forking points, Gaussian-kernel-weighted Dirichlet pooling smooths within segments, and cross-validation tunes the penalty and bandwidth. Smoothing S=5 data multiplies effective sample size by 5.0× at 4-token spacing for Llama (22.1× at 1-token spacing) and 4.9× for DeepSeek sentence resampling, falling to about 1× at S=100. Combined with coarser resampling, it cuts total budget to 1/8 with small error increase, roughly 1% of reference cost in the Figure 1 example. Gains shrink at large forks: at a 0.20 threshold smoothing is marginally worse than raw data. The implication is that resampling-based analysis of reasoning should be treated as a statistical estimation problem with a known noise model, not a brute-force sampling problem, and that sampling budgets can be allocated accordingly.
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse Minkowski sum of low-dimensional manifolds. The paper introduces block-sparse featurizers (BSFs) in three variants: Vanilla (top-k blocks by norm, untied encoder), Grassmannian (tied encoder with orthonormal charts), and Group Lasso (block soft-threshold). On a synthetic manifold-superposition toy, all three reach per-block R² of 0.93–0.97 against an oracle of 0.99, while a TopK SAE collapses to 0.53. On DINOv3 ViT-B, a minimum-description-length analysis at the 20% distortion floor shows blocks of dimension 2–4 describe activations in fewer bits than b=1 SAEs at every dictionary width and sparsity, and stable rank saturates near 3 even when b=16 is allowed. In InceptionV1, the curve-detector neurons and SAE atoms are fragments of one curve manifold, whose first three Fourier harmonics carry 89% of variance (59/18/12%) and survive a matched null by factors of 470 and 314. A DINOv3 block tracks sun luminance and shadow across twelve Blender models, and SDXL blocks support manifold steering, as in the pretzel block. The implication is that a featurizer is a hypothesis about the data-generating process, so interpretability should infer what the atoms must be rather than assume them. Blocks are one answer, matched to vision but probably not to language.
Causal intervention methods central to mechanistic interpretability—including activation patching, mean-difference vector patching, Sparse Autoencoders, and Distributed Alignment Search (DAS)—systematically produce representations that diverge from the target model's natural distribution, and this divergence can corrupt mechanistic conclusions even when behavioral accuracy appears unaffected. For any manifold geometry other than axis-aligned hyperrectangles, coordinate patching is provably guaranteed to produce off-manifold representations given exhaustive sampling, and empirical measurements using Earth Mover's Distance (EMD) confirm divergence across all three tested methods on Meta-Llama-3-8B-Instruct. Two mechanistically distinct failure modes emerge: 'harmless' divergences confined to the behavioral null-space of downstream weight matrices, and 'pernicious' divergences that activate hidden computational pathways or trigger dormant behavioral changes—illustrated concretely with a ReLU circuit where mean-difference patching recruits a third hidden unit silent under all natural class inputs. To mitigate pernicious divergence, the paper applies and modifies the Counterfactual Latent (CL) loss from Grant (2025), showing it reduces EMD from 0.032 ± 0.003 to 0.007 ± 0.001 in synthetic DAS settings while maintaining IIA of 0.997–0.9988, and that training EMD anti-correlates with OOD IIA (coef. −0.34, R² = 0.73, F(1,28) = 75.28, p < 0.001) in a 7B LLM Boundless DAS setting. The paper argues this implies that any divergence outside the null-space of NN layers is potentially pernicious, posing fundamental challenges for aspirations of complete mechanistic understanding using current causal intervention methods alone.
Model Alignment Search (MAS) establishes bidirectional causal similarity between neural networks by learning a per-model orthogonal rotation matrix that isolates behaviorally relevant subspaces and uses interchange interventions — patching those subspaces across frozen model pairs — to measure functional alignment via Interchange Intervention Accuracy (IIA). Comparing GRUs and 2-layer Transformers on numeric tasks reveals that correlative methods like RSA and CKA give misleading estimates: RSA shows anomalously low embedding-layer similarity between same-architecture GRU seeds, and both CKA and RSA suggest potentially high hidden-state similarity between GRU and Transformer hidden states that MAS correctly diagnoses as low because Transformers employ an anti-Markovian solution that recomputes numeric information at every step. MAS compresses behaviorally relevant information to as few as 4 dimensions while achieving IIA comparable to DAS, and it reduces the number of required comparison matrices from O(n²) to O(n), making it more compute-efficient than traditional model stitching for three or more models. A case study on DeepSeek-R1-Distill-Qwen-1.5B models fine-tuned on toxic versus nontoxic text demonstrates that toxic-to-toxic MAS IIA is measurably higher than toxic-to-nontoxic IIA, whereas nontoxic-to-nontoxic comparisons show no significant internal difference — suggesting MAS can serve as a diagnostic for representational misalignment. The Counterfactual Latent MAS (CLMAS) extension, which adds an auxiliary L2 plus cosine loss against prerecorded latent vectors, recovers causal alignment even when one model is causally inaccessible, implying the method may generalize to ANN–biological neural network comparisons where only recordings, not interventions, are available.
More papers — OpenAlex / S2
Studies (1)
Affiliations (2)
- Stanford University(institute)
- PDP Lab(institute)
Co-authors (12)
- Atticus Geiger15 shared
- Ekdeep Singh Lubana15 shared
- Jack Merullo15 shared
- Can Rager13 shared
- Tal Haklay13 shared
- Thomas McGrath13 shared
- Owen Lewis10 shared
- Alexa R. Tartaglini9 shared
- Amir Zur9 shared
- Christopher Potts9 shared
- Simon Jerome Han9 shared
- Dron Hazra6 shared
Their work is cited by (2)
Other inbound relations (1)
Recent mentions (3)
- papers-typedbigelow-2026-forking-fast.md
- papers-typedgrant-2025-addressing-divergent.md
- papers-typedgrant-2025-alignment-search.md