When task gradient norms differ greatly, large-norm tasks have not converged while small-norm tasks have nearly converged

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Task balancing requires simultaneous consideration of both loss scales and gradient magnitudesclaim0.788
  Core interpretive position of DB-MTL: complementarity of loss and gradient perspectives
• We hypothesize that degraded generalization on benchmarks like MMLU may reflect the computational demands of the tasks.hypothesis0.771
  Connecting the paper's task-difficulty findings to prior observations of weak generalization on complex QA benchmarks.
• For a given task, the number of all sequences which work is tiny by comparison with the huge number of all possible sequences; less than a trillionth of all 6 × 10^23 possible sequences actually work well enough.claim0.770
  A combinatorial argument that good sequences are astronomically rare, emphasizing the difficulty of discovery.
• There are fewer representations competent for N tasks than M<N tasks, so training more general models should yield fewer possible solutionshypothesis0.767
  Selective pressure toward convergence via task generality
• Turner et al. 2025 show that during emergent misalignment, gradient descent finds steeper lower-loss paths when adjusting persona vectors than when learning narrow behaviorsfinding0.760
  Mechanistic explanation of why fine-tuning shifts persona vectors rather than directly learning narrow behaviors
• The proposed gradient-magnitude balancing method consistently outperforms GradNorm, as it guarantees equal gradient magnitudes and considers update magnitude.claim0.760
  Advantage over GradNorm.
• Setting αk as the maximum gradient norm among tasks performs best.claim0.760
  Recommended strategy for gradient normalization.
• Maximum gradient norm scalingconcept0.753
  Scaling aggregated gradient by the maximum gradient norm among tasks.