arXiv Machine Learning
Aug 6
Robustness Emerges Early in Training Dynamics, but Is Not Preserved
arXiv:2608. 04442v1 Announce Type: new Abstract: Robustness to natural corruptions remains a fundamental challenge for deep neural networks.
GradientStabilizer:Fix the Norm, Not the Gradient
arXiv:2502. 17055v5 Announce Type: replace Abstract: Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence.
Related stories
arXiv AI
Jun 2
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
arXiv:2606. 00888v1 Announce Type: cross Abstract: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.
arXiv Machine Learning
Jun 2
GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
arXiv:2606. 00539v1 Announce Type: new Abstract: Training stability is a key bottleneck in low-precision language model training: efficient low-cost paths can still produce short-lived numerical risks at a small set of operators.
arXiv Machine Learning
Jun 10
AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
arXiv:2502. 11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining.
arXiv Machine Learning
Jul 8
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.
arXiv Machine Learning
Aug 4
AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.
arXiv Machine Learning
1d ago
Adaptive Optimization via Momentum on Variance-Normalized Gradients
arXiv:2602. 10204v2 Announce Type: replace Abstract: We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization.
arXiv AI
Jul 21
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
arXiv:2605. 29547v2 Announce Type: replace-cross Abstract: Deep learning optimization relies heavily on the assumption of smooth loss landscapes, a condition systematically violated by modern architectures due to non-smooth components such as ReLU activations and quantization operators.
arXiv Machine Learning
Jul 3
ZENITH: Automated Gradient Norm Informed Stochastic Optimization
arXiv:2601. 15212v2 Announce Type: replace Abstract: Training deep computer vision models requires manual oversight or hyperparameter tuning of the learning rate (LR) schedule.
arXiv AI
Jun 11
A Physics-Inspired Optimizer: Velocity Regularized Adam
arXiv:2505. 13196v3 Announce Type: replace-cross Abstract: We introduce Velocity-Regularized Adam (VRAdam), a physics-inspired optimizer for training deep neural networks that draws on ideas from quartic terms for kinetic energy with its stabilizing effects on various system dynamics.
arXiv Machine Learning
Jun 2
Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
arXiv:2412. 19444v2 Announce Type: replace Abstract: Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process.
arXiv Machine Learning
Jul 1
SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates
arXiv:2606. 30930v1 Announce Type: cross Abstract: Modern deep learning has been shown to operate at the edge of stability, routinely using learning rates far larger than those justified by classical optimization theory.