claim
active
claim:once-common-task-features-are-learned-their-gradient-updates-become-weak-leaving-spare-width-for-rare-tasksOnce common-task features are learned, their gradient updates become weak, leaving spare width for rare tasks
Mechanism from Theorem 4.
Source paper
extracted_from(2026) · Jing Huang · Daniel Wurgaft · Rachit Bansal · Laura Ruis +5
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (2)
finding
- Fig. 3 and Fig. 19.
- Fig. 19.
Claims (1)
claim
- Data-induced competition over resources.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Fig. 4 caption.
- We hypothesize that degraded generalization on benchmarks like MMLU may reflect the computational demands of the tasks.hypothesis0.788Connecting the paper's task-difficulty findings to prior observations of weak generalization on complex QA benchmarks.
- When task gradient norms differ greatly, large-norm tasks have not converged while small-norm tasks have nearly convergedhypothesis0.787Motivates setting αk = max norm to enable further learning on under-converged tasks
- Width buys features in order of utility, so only larger models learn infrequent or complex tasksclaim0.783Interpretation of Theorem 3.
- Scale alone is not the mechanism enabling rare-task learning; multi-task methods can also enable itclaim0.779Clarification in related work.
- Motivation for using sparsity-based dictionary learning on language models
- Mechanistic explanation of why fine-tuning shifts persona vectors rather than directly learning narrow behaviors
- Main proposal.