arXiv:2602. 10204v2 Announce Type: replace Abstract: We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization.
By Francisco Patitucci, Aryan Mokhtari
arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.
By Yao Fu, Chunxia Zhang, Junmin Liu, Yihang Jin, Haishan Ye, Yuanao Yang
VCMM: Variance-Calibrated Momentum for Multimodal Learning proposes a new optimizer that adapts momentum based on modality-specific gradient dynamics. It estimates minibatch noise and temporal drift online, using a Kalman-inspired controller to set modality-specific momentum and applies bias correction for the first moment. Experiments on four multimodal benchmarks show consistent improvements with modest training overhead.
By Zhongjing Gu, Chenyang Huang, Yufa Feng, Chong He, Qinxu Ding, Yiming Cui
arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.
By Da Chang, Ganzhao Yuan
arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.
By Ganzhao Yuan
arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.
By Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana, Gopi Krishna Nedanuri