claim

active

claim:setting-aggregated-gradient-scaling-factor-to-maximum-gradient-norm-performs-best-for-task-balancing

Setting aggregated gradient scaling factor to maximum gradient norm performs best for task balancing

Empirical finding on choice of αk in gradient normalization strategy

Source paper

extracted_from

Dual-Balancing for Multi-Task Learning

(2023) · Baijiong Lin · Weisen Jiang · Feiyang Ye · Yu Zhang +5

Neighborhood — ranked by edge-count

Hypotheses (1)

hypothesis

When task gradient norms differ greatly, large-norm tasks have not converged while small-norm tasks have nearly converged
supports
Motivates setting αk = max norm to enable further learning on under-converged tasks

Communities (3)

community

Dual-balancing multi-task learning
members_of
DB-MTL jointly balances loss scale and gradient magnitude, benchmarked on NYUv2 and Office-31.
Dual balancing multi-task learning
members_of
DB-MTL combines loss-scale and gradient-magnitude balancing, benchmarked across NYUv2, Cityscapes, QM9, and Office datasets.
Gradient norm scaling for multitask learning
members_of
Investigates optimal gradient balancing strategies across tasks, finding maximum gradient norm normalization outperforms alternatives in multitask optimization.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Setting αk as the maximum gradient norm among tasks performs best.claim0.832
Recommended strategy for gradient normalization.
Maximum gradient norm scalingconcept0.827
Scaling aggregated gradient by the maximum gradient norm among tasks.
Task balancing requires simultaneous consideration of both loss scales and gradient magnitudesclaim0.807
Core interpretive position of DB-MTL: complementarity of loss and gradient perspectives
Loss-scale balancing and gradient-magnitude balancing are complementary and combining them achieves the best performance.claim0.796
Ablation conclusion.
The proposed gradient-magnitude balancing method consistently outperforms GradNorm, as it guarantees equal gradient magnitudes and considers update magnitude.claim0.796
Advantage over GradNorm.
gradient-magnitude balancingconcept0.786
Addressing disparity in gradient magnitudes across tasks at the gradient level
Setting αk to the maximum gradient norm performs best among tested strategies on NYUv2 (Figure 6).finding0.770
Sensitivity analysis for gradient normalization scaling factor.
The gradient-magnitude balancing method outperforms GradNorm on NYUv2, Cityscapes, Office-31, Office-Home.finding0.768
Comparison of gradient-magnitude balancing with GradNorm.