claim

active

claim:task-balancing-requires-simultaneous-consideration-of-both-loss-scales-and-gradient-magnitudes

Task balancing requires simultaneous consideration of both loss scales and gradient magnitudes

Core interpretive position of DB-MTL: complementarity of loss and gradient perspectives

Source paper

extracted_from

Dual-Balancing for Multi-Task Learning

(2023) · Baijiong Lin · Weisen Jiang · Feiyang Ye · Yu Zhang +5

Neighborhood — ranked by edge-count

Findings (1)

finding

DB-MTL achieves ∆p = +1.15±0.16 on NYUv2, outperforming all baselines including state-of-the-art
supports
Primary empirical validation on scene understanding task

Communities (3)

community

Dual-balancing multi-task learning
members_of
DB-MTL jointly balances loss scale and gradient magnitude, benchmarked on NYUv2 and Office-31.
Dual balancing multi-task learning
members_of
DB-MTL combines loss-scale and gradient-magnitude balancing, benchmarked across NYUv2, Cityscapes, QM9, and Office datasets.
Multi-task learning gradient balancing
members_of
Methods addressing loss-scale and gradient-magnitude imbalances in multi-task learning, with DB-MTL achieving state-of-the-art results on dense prediction benchmarks like NYUv2.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Loss-scale balancing and gradient-magnitude balancing are complementary and combining them achieves the best performance.claim0.881
Ablation conclusion.
DB-MTL achieves loss-scale balancing by performing logarithm transformation on each task loss, and rescales gradient magnitudes by normalizing all task gradients to comparable magnitudes using the maximum gradient norm.quote0.829
Concise summary of the DB-MTL method from the abstract.
Setting aggregated gradient scaling factor to maximum gradient norm performs best for task balancingclaim0.807
Empirical finding on choice of αk in gradient normalization strategy
Task balancing is still an open problem in multi-task learning.claim0.801
Motivation for the proposed method.
gradient-magnitude balancingconcept0.801
Addressing disparity in gradient magnitudes across tasks at the gradient level
When task gradient norms differ greatly, large-norm tasks have not converged while small-norm tasks have nearly convergedhypothesis0.788
Motivates setting αk = max norm to enable further learning on under-converged tasks
loss-scale balancingconcept0.782
Addressing disparity in loss magnitudes across tasks at the loss level
The proposed gradient-magnitude balancing method consistently outperforms GradNorm, as it guarantees equal gradient magnitudes and considers update magnitude.claim0.777
Advantage over GradNorm.