maximum-norm gradient normalization

Training-free technique normalizing all task gradients to the maximum gradient norm magnitude

Neighborhood — ranked by edge-count

framework

GradNorm
associated_with
Gradient balancing method learning task weights; DB-MTL improves on its approach

concept

gradient-magnitude balancing
implements
Addressing disparity in gradient magnitudes across tasks at the gradient level

method

Exponential Moving Average
implements
Used in DB-MTL to estimate batch gradient expectations dynamically

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Maximum gradient norm scalingconcept0.898
Scaling aggregated gradient by the maximum gradient norm among tasks.
Setting αk as the maximum gradient norm among tasks performs best.claim0.787
Recommended strategy for gradient normalization.
Setting aggregated gradient scaling factor to maximum gradient norm performs best for task balancingclaim0.765
Empirical finding on choice of αk in gradient normalization strategy
Gradient methodmethod0.742
Optimization technique that computes weight changes by following the gradient of an error function; contrasted with evolutionary stochastic search.
Gradient Descentmethod0.741
Used for updating hidden state expectations; provides dynamical process theory testable against neuronal data
Setting αk to the maximum gradient norm performs best among tested strategies on NYUv2 (Figure 6).finding0.739
Sensitivity analysis for gradient normalization scaling factor.
Layer Normalization (Ba et al., 2016b)concept0.727
Layer normalisation used in transformer and in TEM-t position encoding preprocessing.
Weak-to-Strong Generalizationconcept0.726
Proposed approach to elicit strong model capabilities with a weak supervisor, raised as related context for reward hacking risks