thinker:nick-cammarataNick Cammarata
Authored papers (2)
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse Minkowski sum of low-dimensional manifolds. The paper introduces block-sparse featurizers (BSFs) in three variants: Vanilla (top-k blocks by norm, untied encoder), Grassmannian (tied encoder with orthonormal charts), and Group Lasso (block soft-threshold). On a synthetic manifold-superposition toy, all three reach per-block R² of 0.93–0.97 against an oracle of 0.99, while a TopK SAE collapses to 0.53. On DINOv3 ViT-B, a minimum-description-length analysis at the 20% distortion floor shows blocks of dimension 2–4 describe activations in fewer bits than b=1 SAEs at every dictionary width and sparsity, and stable rank saturates near 3 even when b=16 is allowed. In InceptionV1, the curve-detector neurons and SAE atoms are fragments of one curve manifold, whose first three Fourier harmonics carry 89% of variance (59/18/12%) and survive a matched null by factors of 470 and 314. A DINOv3 block tracks sun luminance and shadow across twelve Blender models, and SDXL blocks support manifold steering, as in the pretzel block. The implication is that a featurizer is a hypothesis about the data-generating process, so interpretability should infer what the atoms must be rather than assume them. Blocks are one answer, matched to vision but probably not to language.
The Circuits framework proposes that neural network internals are legible at the level of individual neurons and their weighted connections, advancing three speculative claims: features (directions in activation space) are the fundamental unit, features connect via weights to form interpretable circuits, and analogous features and circuits recur across architectures. Working primarily in InceptionV1, the paper demonstrates these claims through three feature types — curve detectors in layer mixed3b responding to curved boundaries at ~60-pixel radius, high-low frequency detectors serving as object-boundary heuristics, and a pose-invariant dog head detector — and three circuits: the curve detector circuit readable directly off 5×5 convolution weights, a four-layer oriented dog-head detection circuit implementing XOR-like inhibition between left- and right-facing pathways before unioning them, and a superposition circuit in mixed4c where a pure car feature is deliberately spread across dog-detector neurons to conserve representable dimensions. The method introduced is the handwritten circuit reimplementation, in which weights are set by hand to reproduce a neuron's function as a falsifiability test. Universality is observed anecdotally across AlexNet, InceptionV1, VGG19, ResNetV2-50, and models trained on Places365, though the authors treat this as preliminary. The paper argues that if all three claims hold, mechanistic interpretability can be grounded as a natural science — with circuits serving as falsifiable, small-scope epistemic units that could eventually compose into full accounts of model behavior, analogous to cellular biology's role in zoology.
More papers — OpenAlex / S2
Studies (2)
Affiliations (1)
- OpenAI(institute)
Co-authors (12)
- Atticus Geiger6 shared
- Dron Hazra6 shared
- Ekdeep Singh Lubana6 shared
- Jack Merullo6 shared
- Lee Sharkey6 shared
- Matthew Kowal6 shared
- Mozes Jacobs6 shared
- Thomas Fel6 shared
- Thomas Serre6 shared
- Usha Bhalla6 shared
- Aiden Swann4 shared
- Can Rager4 shared
Their work is cited by (2)
Other inbound relations (1)
- citesMichael Edward Johnson(thinker)
Recent mentions (5)
- papers-typedolah-2020-introduction.md
- papers
towards.md - papers-typedanima-labs-phenomenology-pt1.md
- papers-typedjohnson-vasocomputation-2023.md
- papers-typedjohnson-vasocomputation-2023.md