arXiv Machine Learning August 4, 2026 By Tianjin Huang, Zhangyang Wang, Haotian Hu, Zhenyu Zhang, Gaojie Jin, Xiang Li, Li Shen, Jiaxing Shang, Tianlong Chen, Ke Li, Lu Liu, Qingsong Wen, Shiwei Liu

GradientStabilizer:Fix the Norm, Not the Gradient

Read the original on arXiv Machine Learning →

arXiv:2502. 17055v5 Announce Type: replace Abstract: Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 6

Robustness Emerges Early in Training Dynamics, but Is Not Preserved

arXiv:2608. 04442v1 Announce Type: new Abstract: Robustness to natural corruptions remains a fundamental challenge for deep neural networks.

By Jiangang Yang, Wenhui Shi, Lu Hu, Jing Xing, Jian Liu

computer-vision benchmarks

arXiv AI

Jun 2

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

arXiv:2606. 00888v1 Announce Type: cross Abstract: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.

By Qiao Xiao, Boqian Wu, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Elena Mocanu, Mykola Pechenizkiy, Decebal Constantin Mocanu, Torsten Hoefler

llms efficiency

arXiv Machine Learning

Jun 2

GNMR: Runtime Stability Control for Low-Precision Large Language Model Training

arXiv:2606. 00539v1 Announce Type: new Abstract: Training stability is a key bottleneck in low-precision language model training: efficient low-cost paths can still produce short-lived numerical risks at a small set of operators.

By Boao Kong, Weichen Jia, Engao Zhang, Guohong Li, Yonghan Dong, Yao Wang, Yaoyuan Wang, Yunke Peng, Kun Yuan

llms fine-tuning efficiency

arXiv Machine Learning

Jun 10

AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping

arXiv:2502. 11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining.

By Guoxia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Dianhai Yu, Yanjun Ma, Li Shen

llms

arXiv Machine Learning

Jul 8

Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.

By Yao Fu, Chunxia Zhang, Junmin Liu, Yihang Jin, Haishan Ye, Yuanao Yang

benchmarks

arXiv Machine Learning

Aug 4

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.

By Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana, Gopi Krishna Nedanuri

benchmarks

Related stories

Robustness Emerges Early in Training Dynamics, but Is Not Preserved

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

GNMR: Runtime Stability Control for Low-Precision Large Language Model Training

AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping

Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization