thinker:andrew-lampinenAndrew Lampinen
Cited for work on representation biases and the call for causal methods in neural comparisons.
Authored papers (1)
Larger models learn tasks smaller models never learn, and the cause is data-induced competition for neurons, not expressivity: small models spend capacity on frequent or simple tasks, and their updates overwrite rare-task features before the next rare observation arrives. The argument starts from power-law scaling. With a compute-optimal exponent γ=0.34 and an asymptotic parameter exponent α≈0.46, α>γ implies that a smaller model's infinite-data loss still exceeds a larger model's finite-compute loss. The paper formalizes this as 'learnable via data scaling' versus 'learnable via model scaling' (Definitions 1–2). In a synthetic mixture of K=32 linear-regression tasks, Theorem 3 shows a width-N encoder keeps the N features with the highest utility π_kλ_kj. Theorem 4 and Proposition 6 show that once common-task residual falls, rare-task features stop being overwritten. A matched-frequency injection protocol isolates retention from frequency. The same pattern appears in OLMo models (4M–4B parameters) pretrained on Dolma v1.7 with injected comparison and modular-addition tasks at frequencies from 7.8×10⁻³ to 2.4×10⁻⁸. Only larger models learn the rare tasks. They carry more task features, such as a 1-D global-order subspace with 96% DAS interchange success. They also show near-orthogonal non-task gradients (7.58×10⁻⁵ at 1B versus 0.10 at 20M). The implication is that scaling is partly a data-mixture problem: raising a target task's frequency may substitute for adding parameters.
More papers — OpenAlex / S2
Affiliations (1)
- Anthropic(institute)
Co-authors (12)
- Christopher Potts3 shared
- Daniel Wurgaft3 shared
- David Alvarez-Melis3 shared
- Ekdeep Singh Lubana3 shared
- Jing Huang3 shared
- Laura Ruis3 shared
- Naomi Saphra3 shared
- Rachit Bansal3 shared
- Andrew K. Lampinen2 shared
- Atticus Geiger1 shared
- Benjamin Edelman1 shared
- Eric Michaud1 shared
Other inbound relations (1)
Recent mentions (2)
- papers-typedhuang-2026-larger-learn.md
- papers-typedgrant-2025-alignment-search.md