finding

active

finding:the-gradient-magnitude-balancing-method-outperforms-gradnorm-on-nyuv2-cityscapes-office-31-office-home

The gradient-magnitude balancing method outperforms GradNorm on NYUv2, Cityscapes, Office-31, Office-Home.

Comparison of gradient-magnitude balancing with GradNorm.

Source paper

extracted_from

Dual-Balancing for Multi-Task Learning

(2023) · Baijiong Lin · Weisen Jiang · Feiyang Ye · Yu Zhang +5

Neighborhood — ranked by edge-count

Claims (1)

claim

The proposed gradient-magnitude balancing method consistently outperforms GradNorm, as it guarantees equal gradient magnitudes and considers update magnitude.
restatessupports
Advantage over GradNorm.

Communities (2)

community

Dual-balancing multi-task learning
members_of
DB-MTL jointly balances loss scale and gradient magnitude, benchmarked on NYUv2 and Office-31.
Gradient magnitude balancing for multitask learning
members_of
Methods that equalize gradient magnitudes across tasks to improve multitask optimization, outperforming GradNorm on vision and domain adaptation benchmarks.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

gradient-magnitude balancingconcept0.819
Addressing disparity in gradient magnitudes across tasks at the gradient level
Loss-scale balancing and gradient-magnitude balancing are complementary and combining them achieves the best performance.claim0.808
Ablation conclusion.
DB-MTL has similar per-epoch running time to gradient balancing methods on NYUv2, slower than loss balancing methods.finding0.793
Computational efficiency comparison.
Combining loss-scale and gradient-magnitude balancing achieves Δp = +1.15±0.16 on NYUv2.finding0.788
Full DB-MTL ablation result.
We find that the logarithm transformation also benefits existing gradient balancing methods.quote0.777
Key finding showing the broader utility of the log transformation.
Setting aggregated gradient scaling factor to maximum gradient norm performs best for task balancingclaim0.768
Empirical finding on choice of αk in gradient normalization strategy
The logarithm transformation also benefits existing gradient balancing methods.claim0.767
Generalization of the loss transformation.
Setting αk to the maximum gradient norm performs best among tested strategies on NYUv2 (Figure 6).finding0.759
Sensitivity analysis for gradient normalization scaling factor.

Restated by (1)

cosine ≥ 0.90

Other entities that say roughly the same thing. May be merge candidates or independent restatements across papers.

claim
The proposed gradient-magnitude balancing method consistently outperforms GradNorm, as it guarantees equal gradient magnitudes and considers update magnitude.