paper
active
2026
paper:doi-10-48550-arxiv-2605-29548

Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention

TL;DR

Larger models learn tasks smaller models never learn, and the cause is data-induced competition for neurons, not expressivity: small models spend capacity on frequent or simple tasks, and their updates overwrite rare-task features before the next rare observation arrives. The argument starts from power-law scaling. With a compute-optimal exponent γ=0.34 and an asymptotic parameter exponent α≈0.46, α>γ implies that a smaller model's infinite-data loss still exceeds a larger model's finite-compute loss. The paper formalizes this as 'learnable via data scaling' versus 'learnable via model scaling' (Definitions 1–2). In a synthetic mixture of K=32 linear-regression tasks, Theorem 3 shows a width-N encoder keeps the N features with the highest utility π_kλ_kj. Theorem 4 and Proposition 6 show that once common-task residual falls, rare-task features stop being overwritten. A matched-frequency injection protocol isolates retention from frequency. The same pattern appears in OLMo models (4M–4B parameters) pretrained on Dolma v1.7 with injected comparison and modular-addition tasks at frequencies from 7.8×10⁻³ to 2.4×10⁻⁸. Only larger models learn the rare tasks. They carry more task features, such as a 1-D global-order subspace with 96% DAS interchange success. They also show near-orthogonal non-task gradients (7.58×10⁻⁵ at 1B versus 0.10 at 20M). The implication is that scaling is partly a data-mixture problem: raising a target task's frequency may substitute for adding parameters.

What to take away

  1. 1. Combining the compute-optimal exponent γ=0.34 with the asymptotic parameter exponent α≈0.46 (Chinchilla-like), α>γ implies a smaller model's infinite-data loss stays above a larger model's finite-compute loss, so some of the distribution is learnable only via model scaling.
  2. 2. Theorem 3 shows the optimal width-N encoder spans the top-N eigenspace of M=Σπ_kC_k, keeping the N features with largest utility u_kj=π_kλ_kj, so rare or slowly-decaying tasks are learned only at larger widths.
  3. 3. In the rank-1 test with K=32 tasks, D=1024 and β=2, the empirical transition width N_emp(k) sat on the predicted staircase N_crit(k)=k across the width grid (1 to 64).
  4. 4. Proposition 6 gives a rare feature a critical width: it is stable against the common-task solution iff π_rλ_r<μ^F_N, so a rare feature displaces the weakest occupied common direction only once the frequent-task residual falls below δ*_F(N_r^crit).
  5. 5. The matched-frequency injection protocol withholds the rare task for G steps and then reinjects it in a batch of m=G·B·ρ_r samples, holding long-run frequency fixed; rare-task signal decays with G far more steeply in narrow models (e.g., N=32) than in wide ones (N=256).
  6. 6. In OLMo models (4M, 20M, 300M, 1B, 4B) trained on Dolma v1.7 with 100-token-set injected tasks, only larger models learned comparison and modular-addition tasks at frequencies as low as 2.4×10⁻⁷, and only large models at higher frequency showed grokking on modular addition.
  7. 7. Distributed alignment search located a 1-D first-layer subspace encoding global token order for the comparison task with a 96% interchange-intervention success rate, and larger models and higher frequencies expressed it faster.
  8. 8. At injection steps, batch-to-task-reference gradient cosine similarity was 0.08±0.02 at 1B versus 0.04±0.04 at 300M; non-task gradient alignment was 0.10±0.09 at 20M but 7.58×10⁻⁵ at 1B, which the paper reads as little interference at 1B.
  9. 9. Extending synthetic training from 100K to 1M steps left above-capacity tasks at or below the mean-predictor baseline, so the learning gap is not just undertraining.
  10. 10. The paper raises the open question of whether frequent tasks undergo an opposite generalize-to-memorize overfit dynamic, and whether scaling avoids it through something like double descent.

Peer brief — for seminar discussion

This work asks why larger models learn tasks that smaller ones do not, even with unlimited data. It starts from a phenomenological argument: with the compute-optimal exponent γ=0.34 and the asymptotic parameter exponent α≈0.46, the asymptotic loss of a small model is still worse than the finite-compute loss of a larger one. It then defines learning via data scaling versus learning via model scaling. To test this, it builds a synthetic multi-task linear-regression setup, K=32 tasks with power-law frequencies and spectra, and a shared width-N bottleneck encoder. Theorem 3 predicts that width N retains the top-N features ranked by utility π_kλ_kj. Theorem 4 and Proposition 6 predict that rare-task learning begins once common-task residual drops below a threshold, because common-task gradients then weaken and stop overwriting rare features. A matched-frequency injection protocol is the central instrument. It holds long-run frequency fixed while varying the gap between rare observations, which separates retention from frequency. The predicted staircase is confirmed in a rank-1 test. A 1M-step run shows the gap does not close with more training. The real-LLM check pretrains OLMo models from 4M to 4B on Dolma v1.7 with injected comparison and modular-addition tasks. Only larger models learn rare instances. They embed more task features (96% DAS success on a 1-D order subspace) and show lower non-task gradient alignment (7.58×10⁻⁵ at 1B versus 0.10 at 20M). The load-bearing finding is that capacity competition and retention, not expressivity, gate rare-task learning. The implication is that raising the frequency of a target task, or reshaping the data mixture, may substitute for parameters. A stated hypothesis is that rare-task learning is an update-and-forget loop in small models. Memorization is also recast as a stepping stone to abstraction, in line with Feldman. A critical reader would push on scope and inference. The OLMo evidence rests on two three-token synthetic tasks, one of them modular addition, with frequencies chosen to match reference tasks. Gradient interference is measured on a handful of first-layer neurons. The asymptotic claim leans on power-law scaling that the paper itself notes need not hold asymptotically. Model comparisons are also confounded because width, depth, and heads change together. An alternative to the injection design would be a scaling-law fit across natural long-tail tasks or a causal ablation of the interference mechanism. Neither is attempted, so the evidence is correlational. Whether this explains natural capability emergence at frontier scale remains unshown, as the limitations section concedes.

Methods (7)

Frameworks (6)

Datasets (3)

Original abstract (expand)

Larger models learn tasks smaller models do not. What drives this phenomenon? We develop a simple phenomenological argument that power-law scaling already suggests that a larger model will be able to learn a part of the data distribution that a smaller model fails to learn, even with infinite training data. To validate this claim and identify its causes, we study the effects of model scaling on a synthetic setup consisting of a mixture of tasks that show monotonic scaling curves. The results point to a data-induced competition over resources (neurons). Specifically, smaller models allocate their neurons to high frequency or low complexity tasks, and so they learn solutions that perform poorly on rare and complex tasks. Moreover, this happens even when solutions capable of expressing the desired task exist. We then assess how a larger model circumvents this data-centric bottleneck, finding that it traces to a reduced interference mechanism: larger models can allocate enough resources to common tasks that the gradient updates for those tasks become weak, which means that they do not overwrite rare-task features as they slowly accumulate. Finally, to further validate these claims, we pretrain OLMo models (4M to 4B parameters) on novel tasks of varying frequency and complexity. The results mirror those from our synthetic data experiments: only the larger OLMo models learn the infrequent and complex tasks, and these larger models embed more task features in their representations and show less gradient interference between tasks. Overall, we offer a data-centric account of why larger models learn tasks that smaller models fail to. This helps explain why larger models are better in practice, and it can inform practical questions concerning model sizing and training data mixtures.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+21 more

Similar preprints — Semantic Scholar