arXiv Machine Learning August 4, 2026 By Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana, Gopi Krishna Nedanuri

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

Read the original on arXiv Machine Learning →

arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

benchmarks

arXiv Machine Learning

Jun 17

MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization

arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.

By Da Chang, Ganzhao Yuan

llms fine-tuning safety

arXiv AI

Aug 6

MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training

arXiv:2608. 04407v1 Announce Type: cross Abstract: Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable.

By Masato Fujitake

llms

arXiv Machine Learning

Jun 10

AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping

arXiv:2502. 11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining.

By Guoxia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Dianhai Yu, Yanjun Ma, Li Shen

llms

arXiv Machine Learning

Jul 31

Towards Stability of Parameter-Free Optimization

arXiv:2405. 04376v4 Announce Type: replace Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge.

By Yijiang Pang, Shuyang Yu, Bao Hoang, Jiayu Zhou

arXiv Machine Learning

Aug 4

GradientStabilizer:Fix the Norm, Not the Gradient

arXiv:2502. 17055v5 Announce Type: replace Abstract: Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence.

By Tianjin Huang, Zhangyang Wang, Haotian Hu, Zhenyu Zhang, Gaojie Jin, Xiang Li, Li Shen, Jiaxing Shang, Tianlong Chen, Ke Li, Lu Liu, Qingsong Wen, Shiwei Liu

llms reinforcement-learning efficiency

arXiv Machine Learning

Jul 13