thinker
active
thinker:eric-michaud

Eric Michaud

Identified linebreaking as a top-400 quanta of model behavior.

Authored
5
Introduces
0
Studies
0
Affiliations
2
Cited by
10

Authored papers (5)

  • Claude 3.5 Haiku's linebreaking in fixed-width text is implemented by manipulating low-dimensional, rippled 1-D feature manifolds: character count, line width, characters remaining and next-token length are each embedded with high curvature in small residual-stream subspaces, and the model computes with them through linear geometric operations. Character count lives on a curve in a 6-dimensional subspace (95% of variance of 150 mean activations at layer 2), parametrized by 10 crosscoder features from a 10M-feature Weakly Causal Crosscoder. Attention 'boundary heads' twist the count manifold through QK so count i aligns with line width k=i+ε, raising probe cosine similarity from ~0.25 in the residual stream to ≈1 in QK space. Several heads with different offsets tile 'characters remaining'. Counts for characters remaining and next-token length sit in near-orthogonal subspaces, making the break decision linearly separable (AUC 0.91). The curvature is built by many attention heads, with 5 layer 0 heads reaching R² 0.93 and 11 heads in layers 0-1 reaching 0.97. The paper pairs attribution graphs and feature dictionaries with explicit manifold parametrization, and validates with subspace ablations, mean-activation interventions and '@@' visual illusions. Ringing is shown to be optimal under a capacity-versus-resolution tradeoff. Dictionary features alone impose a complexity tax that geometry reduces, and natural pretraining tasks are good targets for mechanistic biology, though parametrizing manifolds remains manual.

  • Larger models learn tasks smaller models never learn, and the cause is data-induced competition for neurons, not expressivity: small models spend capacity on frequent or simple tasks, and their updates overwrite rare-task features before the next rare observation arrives. The argument starts from power-law scaling. With a compute-optimal exponent γ=0.34 and an asymptotic parameter exponent α≈0.46, α>γ implies that a smaller model's infinite-data loss still exceeds a larger model's finite-compute loss. The paper formalizes this as 'learnable via data scaling' versus 'learnable via model scaling' (Definitions 1–2). In a synthetic mixture of K=32 linear-regression tasks, Theorem 3 shows a width-N encoder keeps the N features with the highest utility π_kλ_kj. Theorem 4 and Proposition 6 show that once common-task residual falls, rare-task features stop being overwritten. A matched-frequency injection protocol isolates retention from frequency. The same pattern appears in OLMo models (4M–4B parameters) pretrained on Dolma v1.7 with injected comparison and modular-addition tasks at frequencies from 7.8×10⁻³ to 2.4×10⁻⁸. Only larger models learn the rare tasks. They carry more task features, such as a 1-D global-order subspace with 96% DAS interchange success. They also show near-orthogonal non-task gradients (7.58×10⁻⁵ at 1B versus 0.10 at 20M). The implication is that scaling is partly a data-mixture problem: raising a target task's frequency may substitute for adding parameters.

  • Sparse autoencoder scaling laws can break down when activations contain feature manifolds: if per-feature loss falls slowly with latent count, an SAE spends its latents tiling common manifolds instead of discovering rarer features. Michaud, Gorton and McGrath adapt Brill's (2024) capacity-allocation model to SAEs. The model casts SAE training as allocating N latents across features with Zipfian frequencies p_i ∝ i^-(1+α) and per-feature loss L_i(n_i) ∝ n_i^-β. It predicts two regimes. When α < β, loss scales as N^-α and discovered features D(N) grow proportionally to N. When β < α, loss scales as N^-β and D(N) ∝ N^((1+β)/(1+α)), so D(N)/N goes to 0. In a simulation where one manifold has β ≪ α, an SAE with 100 million latents discovers only 3 million features. ReLU L1 SAEs on synthetic hyperspheres show β ≈ 0.05 for dimensions 6–8, which is pathological. Once radial variation is added, loss plateaus at roughly n_i ≈ 2d_i latents, a basis solution. Gemma Scope layer-12 latent frequency slopes of −0.57 to −0.74 suggest α ≈ 0.5–0.7. The paper does not settle whether real SAEs are pathological. It argues that the risk depends on manifold geometry, and that realistic radial variation may protect against it. It presents the work as a framing of the problem, not an answer.

More papers — OpenAlex / S2

Affiliations (2)