The paper introduces Activation-Keyed Momentum (AK‑Momentum), a momentum update that uses the input activation of a linear layer as a key to apply a delta‑rule update, allowing each direction to decay at a rate proportional to its frequency of appearance. AK‑Momentum is proven to be a valid momentum, incorporates input‑side curvature correction without matrix inversion, and clears stale directions faster than traditional exponential moving average (EMA) under both fixed and drifting optima. It can replace the momentum buffer of any optimizer, scales with width under μP, adds only 22–25% extra compute, and demonstrates significant step‑count reductions in FineWeb‑Edu pretraining and other benchmarks.
whyItMatters":"AK‑Momentum offers a principled, efficient way to adapt momentum decay to anisotropic training dynamics, improving convergence speed and stability across a range of models and datasets."
By Euijin Hong, Guannan Qu
arXiv:2608. 19491v1 Announce Type: new Abstract: Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate.
By Euijin Hong, Guannan Qu
arXiv:2601.12971v2 Announce Type: replace
Abstract: Physics-informed neural networks (PINNs) can be limited by coordinate representations and conflicting gradients from heterogeneous physical constra...
By Pancheng Niu, Jun Guo, Qiaolin He, Yongming Chen, Yanchao Shi
The paper introduces PAM-GS, a physics-aware gradient surgery technique for Physics-Informed Neural Networks (PINNs). It addresses the highly imbalanced multi-task optimisation problem in PINNs by adaptively mitigating task interference based on observed gradient conflicts. Experiments on four PDE benchmarks show that PAM-GS achieves competitive solution accuracy while maintaining strong task-balanced performance, outperforming existing methods on most problems.
By Thomas Borsani, Giuseppe Di Fatta
arXiv:2610. 11475v1 Announce Type: new Abstract: Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining.
By Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai, Jiaqi Wu, Chenyu Zhu, Tong Che
arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.
By Da Chang, Ganzhao Yuan
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.
By Alexander H\"agele, Alejandro Hern\'andez-Cano, Atli Kosson, Martin Jaggi
arXiv:2607. 07156v1 Announce Type: new Abstract: Different optimizers have different update biases, but these biases are usually implicit.
By Zhang Gongyue, Liu Donghan, Ren Weihong, Sheng Yixuan, Wang Zhiyong, Liu Honghai
arXiv:2605. 23391v2 Announce Type: replace Abstract: Physics-informed neural networks (PINNs) for coupled multiphysics systems suffer systematic accuracy degradation as inter-equation coupling strengthens.
By Youngjae Park, Jaemin Kim, Junghwa Hong
Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict finite-batch stochastic gradient descent (SGD) qualitatively, and we trace the discrepancy to a specific mechanism.
arXiv:2608.08322v2 Announce Type: replace-cross
Abstract: Adaptive loss-balancing schemes for physics-informed neural networks rest on a premise that every residual should be driven to zero. For leve...
By Muhammad Akbar Khan
arXiv:2606. 27354v1 Announce Type: cross Abstract: Neural surrogate models offer fast approximate mappings from PDE parameters to solutions, but they typically treat solving as a purely statistical task: once trained, they struggle to correct their own constraint violations and extrapolate beyond the training distribution.
By Haina Jiang, Liam Wang, Peng-Chen Chen, Min Seop Kwak, Seungryong Kim, Brian Bell, Jeong Joon Park