The paper identifies a problem called Gradient-Update Mismatch (GUM), where optimizers can alter conflict-free gradient directions produced by gradient surgery, leading to conflicts between physics residual and boundary condition losses in Physics-Informed Neural Networks (PINNs). To address this, the authors propose Gradient-Update Alignment (GUA), which projects the optimizer’s update onto the conflict-free cone and adjusts internal optimizer state accordingly. Experiments show GUM is common across many optimizers, and GUA consistently eliminates conflicts and significantly reduces error in PINN training.
arXiv:2608. 19491v1 Announce Type: new Abstract: Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate.
arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.
arXiv:2607. 07156v1 Announce Type: new Abstract: Different optimizers have different update biases, but these biases are usually implicit.
arXiv:2605. 23391v2 Announce Type: replace Abstract: Physics-informed neural networks (PINNs) for coupled multiphysics systems suffer systematic accuracy degradation as inter-equation coupling strengthens.
arXiv:2608.08322v2 Announce Type: replace-cross Abstract: Adaptive loss-balancing schemes for physics-informed neural networks rest on a premise that every residual should be driven to zero. For leve...
arXiv:2606. 27354v1 Announce Type: cross Abstract: Neural surrogate models offer fast approximate mappings from PDE parameters to solutions, but they typically treat solving as a purely statistical task: once trained, they struggle to correct their own constraint violations and extrapolate beyond the training distribution.
arXiv:2607. 14516v1 Announce Type: new Abstract: Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks.
arXiv:2608.29448v1 Announce Type: cross Abstract: Physics-informed neural networks (PINNs) often face ill-conditioned objectives that limit high-accuracy training. Dense quasi-Newton methods improve...
arXiv:2607. 02194v1 Announce Type: new Abstract: Physics-informed neural networks (PINNs) have emerged as a promising route to solve partial differential equations, yet they have struggled to reach the precision of classical solvers.
arXiv:2607. 05271v1 Announce Type: new Abstract: Physics-informed neural networks (PINNs) encounter ill-posed optimization, loss competition, and parameter compensation in partial differential equation (PDE) inverse problems.
arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.