paper:doi-10-48550-arxiv-2605-29548Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
TL;DR
Larger models learn tasks smaller models never learn, and the cause is data-induced competition for neurons, not expressivity: small models spend capacity on frequent or simple tasks, and their updates overwrite rare-task features before the next rare observation arrives. The argument starts from power-law scaling. With a compute-optimal exponent γ=0.34 and an asymptotic parameter exponent α≈0.46, α>γ implies that a smaller model's infinite-data loss still exceeds a larger model's finite-compute loss. The paper formalizes this as 'learnable via data scaling' versus 'learnable via model scaling' (Definitions 1–2). In a synthetic mixture of K=32 linear-regression tasks, Theorem 3 shows a width-N encoder keeps the N features with the highest utility π_kλ_kj. Theorem 4 and Proposition 6 show that once common-task residual falls, rare-task features stop being overwritten. A matched-frequency injection protocol isolates retention from frequency. The same pattern appears in OLMo models (4M–4B parameters) pretrained on Dolma v1.7 with injected comparison and modular-addition tasks at frequencies from 7.8×10⁻³ to 2.4×10⁻⁸. Only larger models learn the rare tasks. They carry more task features, such as a 1-D global-order subspace with 96% DAS interchange success. They also show near-orthogonal non-task gradients (7.58×10⁻⁵ at 1B versus 0.10 at 20M). The implication is that scaling is partly a data-mixture problem: raising a target task's frequency may substitute for adding parameters.
What to take away
- 1. Combining the compute-optimal exponent γ=0.34 with the asymptotic parameter exponent α≈0.46 (Chinchilla-like), α>γ implies a smaller model's infinite-data loss stays above a larger model's finite-compute loss, so some of the distribution is learnable only via model scaling.
- 2. Theorem 3 shows the optimal width-N encoder spans the top-N eigenspace of M=Σπ_kC_k, keeping the N features with largest utility u_kj=π_kλ_kj, so rare or slowly-decaying tasks are learned only at larger widths.
- 3. In the rank-1 test with K=32 tasks, D=1024 and β=2, the empirical transition width N_emp(k) sat on the predicted staircase N_crit(k)=k across the width grid (1 to 64).
- 4. Proposition 6 gives a rare feature a critical width: it is stable against the common-task solution iff π_rλ_r<μ^F_N, so a rare feature displaces the weakest occupied common direction only once the frequent-task residual falls below δ*_F(N_r^crit).
- 5. The matched-frequency injection protocol withholds the rare task for G steps and then reinjects it in a batch of m=G·B·ρ_r samples, holding long-run frequency fixed; rare-task signal decays with G far more steeply in narrow models (e.g., N=32) than in wide ones (N=256).
- 6. In OLMo models (4M, 20M, 300M, 1B, 4B) trained on Dolma v1.7 with 100-token-set injected tasks, only larger models learned comparison and modular-addition tasks at frequencies as low as 2.4×10⁻⁷, and only large models at higher frequency showed grokking on modular addition.
- 7. Distributed alignment search located a 1-D first-layer subspace encoding global token order for the comparison task with a 96% interchange-intervention success rate, and larger models and higher frequencies expressed it faster.
- 8. At injection steps, batch-to-task-reference gradient cosine similarity was 0.08±0.02 at 1B versus 0.04±0.04 at 300M; non-task gradient alignment was 0.10±0.09 at 20M but 7.58×10⁻⁵ at 1B, which the paper reads as little interference at 1B.
- 9. Extending synthetic training from 100K to 1M steps left above-capacity tasks at or below the mean-predictor baseline, so the learning gap is not just undertraining.
- 10. The paper raises the open question of whether frequent tasks undergo an opposite generalize-to-memorize overfit dynamic, and whether scaling avoids it through something like double descent.
Peer brief — for seminar discussion
This work asks why larger models learn tasks that smaller ones do not, even with unlimited data. It starts from a phenomenological argument: with the compute-optimal exponent γ=0.34 and the asymptotic parameter exponent α≈0.46, the asymptotic loss of a small model is still worse than the finite-compute loss of a larger one. It then defines learning via data scaling versus learning via model scaling. To test this, it builds a synthetic multi-task linear-regression setup, K=32 tasks with power-law frequencies and spectra, and a shared width-N bottleneck encoder. Theorem 3 predicts that width N retains the top-N features ranked by utility π_kλ_kj. Theorem 4 and Proposition 6 predict that rare-task learning begins once common-task residual drops below a threshold, because common-task gradients then weaken and stop overwriting rare features. A matched-frequency injection protocol is the central instrument. It holds long-run frequency fixed while varying the gap between rare observations, which separates retention from frequency. The predicted staircase is confirmed in a rank-1 test. A 1M-step run shows the gap does not close with more training. The real-LLM check pretrains OLMo models from 4M to 4B on Dolma v1.7 with injected comparison and modular-addition tasks. Only larger models learn rare instances. They embed more task features (96% DAS success on a 1-D order subspace) and show lower non-task gradient alignment (7.58×10⁻⁵ at 1B versus 0.10 at 20M). The load-bearing finding is that capacity competition and retention, not expressivity, gate rare-task learning. The implication is that raising the frequency of a target task, or reshaping the data mixture, may substitute for parameters. A stated hypothesis is that rare-task learning is an update-and-forget loop in small models. Memorization is also recast as a stepping stone to abstraction, in line with Feldman. A critical reader would push on scope and inference. The OLMo evidence rests on two three-token synthetic tasks, one of them modular addition, with frequencies chosen to match reference tasks. Gradient interference is measured on a handful of first-layer neurons. The asymptotic claim leans on power-law scaling that the paper itself notes need not hold asymptotically. Model comparisons are also confounded because width, depth, and heads change together. An alternative to the injection design would be a scaling-law fit across natural long-tail tasks or a causal ablation of the interference mechanism. Neither is attempted, so the evidence is correlational. Whether this explains natural capability emergence at frontier scale remains unshown, as the limitations section concedes.
Methods (7)
- Data injection frameworkInject special task instances into pretraining data at controlled frequency.
- Fourier mode detectionVariance-fraction test with θ=2 null baseline for modular addition.
- Matched-frequency injection protocolWithhold rare task for G steps then reinject so overall frequency is constant.
- Mixture of linear regression tasks (teacher-student setup)Synthetic multi-task setup with power-law frequencies and spectra.
- MLP-layer null interventionZeroing MLP layers to localize task neurons.
- Normalized Signal metricTask signal in representation normalized to random baseline.
- Task reference direction g_rAggregated task-loss gradient used to measure interference via cosine similarity.
Frameworks (6)
- Chinchilla scaling (compute-optimal)Source of exponents α≈0.46, β≈0.51 and compute estimates.
- Distributed Alignment Search (DAS)Practical method by Geiger et al. for finding distributed causal abstractions using gradient descent
- Lottery ticket hypothesisSurviving subnetworks trainable in isolation.
- Proposition 6: Interference Reduces via ScalingRare direction stable iff πrλr ≥ µ_N^F; defines critical width.
- Theorem 3: Features are Learned in Order of UtilityWidth-N minimizer spans top-N eigenspace of M.
- Theorem 4: Residual Controls LearningCommon-task gradient bounded by residual.
Datasets (3)
- Comparison task T_CMPInjected 3-token comparison task.
- Dolma v1.7Pretraining corpus.
- Modular addition task T_ADDInjected 3-token modular addition task.
Findings (22)
- In matched-frequency injection, rare-task signal decays with gap G at all widths, much more steeply for narrow models, while wide models retain and accumulate signal
Fig. 4.
- With wide complexity range [2,12], learning order is non-monotonic in frequency because a complex task's third mode can outrank frequent tasks' higher modes
Fig. 13.
- Wide toy models show lower gradient cosine similarity between frequent and rare gradients, so frequent updates do not overwrite rare features
Fig. 4 caption.
- At equal general LM loss, larger models achieve lower task loss when frequency is low (2.4e-7); similar trajectories at higher frequency
Fig. 22.
- Task features (token order, Fourier modes) are more present in larger models and higher-frequency runs, and correlate with test accuracy
Fig. 7.
- α>γ (α≈0.46 asymptotic vs γ=0.34 compute-optimal) in practice, so asymptotic loss of larger models exceeds that of smaller ones
Basis of phenomenological argument.
- DAS finds a 1-D subspace in first-layer residual stream encoding global token order with 96% interchange intervention success
App. B.4.
- Larger-width students learn infrequent tasks better in K=32 regression mixture; per-task loss tracks Theorem 3 prediction
Fig. 2 verification.
- OLMo models trained with larger injection gaps N∈{1,10,20,50,100} reach higher task loss despite equal global frequency
Fig. 6b.
- 20M model non-task gradient has cosine similarity 0.10±0.09 with task direction, with wildly oscillating similarity
Fig. 9.
Claims (13)
- Memorization can be a mechanism supporting abstraction: retaining instances longer lets models accumulate signal to learn generalizable structure
Discussion and App. A.
- Power-law scaling already implies a larger model can learn a part of the distribution a smaller model fails to learn even with infinite data
Core phenomenological argument.
- The advantage of larger models is not merely sample efficiency but a more fundamental limitation of smaller models under infinite data
Distinguishes from undertraining.
- Smaller models allocate neurons to high-frequency or low-complexity tasks, leaving poor solutions on rare and complex tasks
Data-induced competition over resources.
- Mere expressivity is not the issue; the question is learnability under gradient optimization from a data mixture
All tasks are learnable by the models.
- Rare-task alignment decays exponentially across the gap between rare observations in a one-neuron two-task model
Example 7.
- Once common-task features are learned, their gradient updates become weak, leaving spare width for rare tasks
Mechanism from Theorem 4.
- Scale alone is not the mechanism enabling rare-task learning; multi-task methods can also enable it
Clarification in related work.
- Frequency alone is insufficient to predict learning order; both frequency and complexity are needed
From complexity sweeps.
- Width buys features in order of utility, so only larger models learn infrequent or complex tasks
Interpretation of Theorem 3.
Hypotheses (1)
- Larger models learn rare tasks via reduced interference and better retention
Hypothesis box in §3.2.
Questions (3)
- if a model sees too many observations of the same task, does it perhaps undergo overfitting, and does scaling help avoid this via double descent?
Future work for frequent tasks.
- what is it about the data that only a larger model can learn, such that the smaller model cannot, even after observing infinite data?
Motivating question.
- How precisely does having more parameters aid this learning?
Motivating question.
Original abstract (expand)
Larger models learn tasks smaller models do not. What drives this phenomenon? We develop a simple phenomenological argument that power-law scaling already suggests that a larger model will be able to learn a part of the data distribution that a smaller model fails to learn, even with infinite training data. To validate this claim and identify its causes, we study the effects of model scaling on a synthetic setup consisting of a mixture of tasks that show monotonic scaling curves. The results point to a data-induced competition over resources (neurons). Specifically, smaller models allocate their neurons to high frequency or low complexity tasks, and so they learn solutions that perform poorly on rare and complex tasks. Moreover, this happens even when solutions capable of expressing the desired task exist. We then assess how a larger model circumvents this data-centric bottleneck, finding that it traces to a reduced interference mechanism: larger models can allocate enough resources to common tasks that the gradient updates for those tasks become weak, which means that they do not overwrite rare-task features as they slowly accumulate. Finally, to further validate these claims, we pretrain OLMo models (4M to 4B parameters) on novel tasks of varying frequency and complexity. The results mirror those from our synthetic data experiments: only the larger OLMo models learn the infrequent and complex tasks, and these larger models embed more task features in their representations and show less gradient interference between tasks. Overall, we offer a data-centric account of why larger models learn tasks that smaller models fail to. This helps explain why larger models are better in practice, and it can inform practical questions concerning model sizing and training data mixtures.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 82%
- What do Language Models Learn and When? The Implicit Curriculum HypothesisKaiser Sun, Millicent Li, Isabelle Lee, Lindia Tjuatja, Jen-tse Huang, Graham Neubig Emmy Liu2026≈ 85%
- One Token Away from Collapse: The Fragility of Instruction-Tuned HelpfulnessSeyedarmin Azizi, Souvik Kundu, Massoud Pedram Erfan Baghaei Potraghloo2026≈ 84%
- Identifying and Mitigating the Influence of the Prior Distribution in Large Language ModelsVeniamin Veselovsky, R. Thomas McCoy, Thomas L. Griffiths Liyi Zhang2025≈ 84%
- Scaling Laws for Downstream Task Performance of Large Language ModelsNatalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, Sanmi Koyejo Berivan Isik2026≈ 84%
- Small LLMs Do Not Learn a Generalizable Theory of Mind via Reinforcement LearningSneheel Sarangi and Hanan Salam2025≈ 84%
- Quantifying LLM Attention-Head Stability: Implications for Circuit UniversalityJack Stanley, Praneet Suresh, Danilo Bzdok Karan Bali2026≈ 84%
- Distributed Specialization: Rare-Token Neurons in Large Language ModelsHaozheng Wang, Yueheng Li Jing Liu2025≈ 83%
- Sparse Semantic Dimension as a Generalization Certificate for LLMsAsif Ekbal Dibyanayan Bandyopadhyay2026≈ 83%
- Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMsAjay Jaiswal, Shiwei Liu, Souvik Kundu, Zhangyang Wang Lu Yin2026≈ 83%
- Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even PerformanceIvan Srba, Maria Bielikova Branislav Pecher2026≈ 83%
- Weight-sparse transformers have interpretable circuitsAchyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, Dan Mossing Leo Gao2025≈ 83%
- Specialization after Generalization: Towards Understanding Test-Time Training in Foundation ModelsPatrik Wolf, Alexander Shevchenko, Dennis J\"uni, Andreas Krause, Gil Kur Jonas H\"ubotter2026≈ 83%
- A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious MinimaHarshvardhan Saini, Zhaoqian Yao, Zheng Lin, Yizhen Liao, Jingyi Cui, Yisen Wang, Mengnan Du, Dianbo Liu Yiming Tang2026≈ 83%
- Probing Representation Forgetting in Supervised and Unsupervised Continual LearningNader Asadi, Sudhir Mudur, Rahaf Aljundi, Eugene Belilovsky MohammadReza Davari2022≈ 83%
- Mechanistically analyzing the effects of fine-tuning on procedurally defined tasksRobert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rockt\"aschel, David Scott Krueger Samyak Jain2024≈ 83%
- ≈ 83%
- Insights on Crosscoder Model Diffingin corpus≈ 82%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 82%
- ≈ 82%
- Toy Models of Superpositionin corpus2022≈ 81%
- ≈ 81%
- ≈ 81%
- Stochastic Parameter Decompositionin corpus2025≈ 80%
- The Platonic Representation Hypothesisin corpus2024≈ 80%
- Learning without neurons in physical systemsin corpus2022≈ 80%
- ≈ 80%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 80%
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?in corpus2025≈ 80%
- ≈ 80%
+21 more