claim
active
claim:power-law-scaling-already-implies-a-larger-model-can-learn-a-part-of-the-distribution-a-smaller-model-fails-to-learn-even-with-infinite-dataPower-law scaling already implies a larger model can learn a part of the distribution a smaller model fails to learn even with infinite data
Core phenomenological argument.
Source paper
extracted_from(2026) · Jing Huang · Daniel Wurgaft · Rachit Bansal · Laura Ruis +5
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (3)
finding
- α>γ (α≈0.46 asymptotic vs γ=0.34 compute-optimal) in practice, so asymptotic loss of larger models exceeds that of smaller onesassociated_withsupportsBasis of phenomenological argument.
- Fig. 23.
- Extending training to 1M steps leaves above-capacity tasks at or below baseline indefinitelysupportsFig. 21.
Concepts (1)
concept
- Emergent Abilities of LLMscontradictsPrior work documenting abrupt capability changes under scale; UCCT provides a measurable predictor for when they occur
Claims (2)
claim
- Width buys features in order of utility, so only larger models learn infrequent or complex tasksextendsInterpretation of Theorem 3.
- Distinguishes from undertraining.
Questions (1)
question
- Motivating question.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Empirically observed power law relationship between data scale and model performance; supports convergence hypothesis
- Cited hypothesis from Lin et al. 2022 suggesting larger models become more capable of deception
- Distinguishes their setting from this one.
- Observation that SAE loss decreases as a power law with compute budget.
- Bigger models are more likely to converge to a shared representation than smaller modelshypothesis0.788Selective pressure toward convergence via model capacity
- Implication of PRH for AI fairness and bias
- Key limitation of the PRH for non-bijective observations
- Future work for frequent tasks.