claim

active

claim:the-magnitude-of-the-normalized-gradients-choice-of-k-plays-an-important-role-in-performance

The magnitude of the normalized gradients (choice of αk) plays an important role in performance.

Insight about gradient normalization scaling.

Source paper

extracted_from

Dual-Balancing for Multi-Task Learning

(2023) · Baijiong Lin · Weisen Jiang · Feiyang Ye · Yu Zhang +5

Neighborhood — ranked by edge-count

Findings (1)

finding

Setting αk to the maximum gradient norm performs best among tested strategies on NYUv2 (Figure 6).
supports
Sensitivity analysis for gradient normalization scaling factor.

Communities (3)

community

Dual-balancing multi-task learning
members_of
DB-MTL jointly balances loss scale and gradient magnitude, benchmarked on NYUv2 and Office-31.
Dual balancing multi-task learning
members_of
DB-MTL combines loss-scale and gradient-magnitude balancing, benchmarked across NYUv2, Cityscapes, QM9, and Office datasets.
Gradient norm scaling for multitask learning
members_of
Investigates optimal gradient balancing strategies across tasks, finding maximum gradient norm normalization outperforms alternatives in multitask optimization.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Setting αk as the maximum gradient norm among tasks performs best.claim0.827
Recommended strategy for gradient normalization.
Gradients play a large role in nature (mountains, electric fields, embryos, rivers) and are mirrored by the success of mathematical tools like tensor calculus.claim0.778
We hypothesize that degraded generalization on benchmarks like MMLU may reflect the computational demands of the tasks.hypothesis0.762
Connecting the paper's task-difficulty findings to prior observations of weak generalization on complex QA benchmarks.
The proposed gradient-magnitude balancing method consistently outperforms GradNorm, as it guarantees equal gradient magnitudes and considers update magnitude.claim0.762
Advantage over GradNorm.
Feature attribution (gradient-based) correlates 0.8 with ablation effects on the 'John' and 'Kobe' examples.finding0.760
Validation of attribution as a fast proxy for causal importance.
Gradients are nearly non-existent in the modern environment because naive standardization, mass-production, and regulation of sizes all work against their formationclaim0.748
Claim that one of the most powerful forms of life has been almost removed from the environment by industrial production norms
The results generalise readily to non-equilibrium systems where scaling relationships remain similar (e.g., dynamic or localised scaling).claim0.739
Claim about broader applicability of the scaling argument
The gradient ∇θL is an inherently signed, directional quantity; a system with access to error magnitude but not directional valence cannot compute itclaim0.738
Mathematical constraint showing that backpropagation requires signed information