Setting αk as the maximum gradient norm among tasks performs best.

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Setting aggregated gradient scaling factor to maximum gradient norm performs best for task balancingclaim0.832
  Empirical finding on choice of αk in gradient normalization strategy
• The magnitude of the normalized gradients (choice of αk) plays an important role in performance.claim0.827
  Insight about gradient normalization scaling.
• Maximum gradient norm scalingconcept0.800
  Scaling aggregated gradient by the maximum gradient norm among tasks.
• maximum-norm gradient normalizationmethod0.787
  Training-free technique normalizing all task gradients to the maximum gradient norm magnitude
• When task gradient norms differ greatly, large-norm tasks have not converged while small-norm tasks have nearly convergedhypothesis0.760
  Motivates setting αk = max norm to enable further learning on under-converged tasks
• Software implementations for all of the models/behaviours presented are common for n = 2, and can be made very efficient for α_i that map many objects onto a much smaller set of object families.claim0.749
  Claim about current practical feasibility and efficiency of 2-way associative implementations.
• The proposed gradient-magnitude balancing method consistently outperforms GradNorm, as it guarantees equal gradient magnitudes and considers update magnitude.claim0.746
  Advantage over GradNorm.
• We hypothesize that degraded generalization on benchmarks like MMLU may reflect the computational demands of the tasks.hypothesis0.744
  Connecting the paper's task-difficulty findings to prior observations of weak generalization on complex QA benchmarks.