← Back to all news
arXiv Machine Learning By Jing Xiao, Xinhai Chen, Qinglin Wang, Menghan Jia, Zhiquan Lai, Dongsheng Li, Jie Liu, Tiejun Li

Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks

Read the original on arXiv Machine Learning →

The paper identifies a problem called Gradient-Update Mismatch (GUM), where optimizers can alter conflict-free gradient directions produced by gradient surgery, leading to conflicts between physics residual and boundary condition losses in Physics-Informed Neural Networks (PINNs). To address this, the authors propose Gradient-Update Alignment (GUA), which projects the optimizer’s update onto the conflict-free cone and adjusts internal optimizer state accordingly. Experiments show GUM is common across many optimizers, and GUA consistently eliminates conflicts and significantly reduces error in PINN training.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 4

Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule

The paper introduces Activation-Keyed Momentum (AK‑Momentum), a momentum update that uses the input activation of a linear layer as a key to apply a delta‑rule update, allowing each direction to decay at a rate proportional to its frequency of appearance. AK‑Momentum is proven to be a valid momentum, incorporates input‑side curvature correction without matrix inversion, and clears stale directions faster than traditional exponential moving average (EMA) under both fixed and drifting optima. It can replace the momentum buffer of any optimizer, scales with width under μP, adds only 22–25% extra compute, and demonstrates significant step‑count reductions in FineWeb‑Edu pretraining and other benchmarks. whyItMatters":"AK‑Momentum offers a principled, efficient way to adapt momentum decay to anisotropic training dynamics, improving convergence speed and stability across a range of models and datasets."

By Euijin Hong, Guannan Qu
More like this →
arXiv Machine Learning
Sep 29

Gradient Surgery for Physics-Informed Neural Networks

The paper introduces PAM-GS, a physics-aware gradient surgery technique for Physics-Informed Neural Networks (PINNs). It addresses the highly imbalanced multi-task optimisation problem in PINNs by adaptively mitigating task interference based on observed gradient conflicts. Experiments on four PDE benchmarks show that PAM-GS achieves competitive solution accuracy while maintaining strong task-balanced performance, outperforming existing methods on most problems.

By Thomas Borsani, Giuseppe Di Fatta
benchmarks
More like this →