arXiv Machine Learning July 27, 2026 By Sharan Sahu, Cameron J. Hogan, Martin T. Wells

On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization

Read the original on arXiv Machine Learning →

arXiv:2601. 12238v5 Announce Type: replace-cross Abstract: In this paper, we provide a comprehensive theoretical analysis of Stochastic Gradient Descent (SGD) and its momentum variants (Polyak Heavy-Ball and Nesterov) for tracking time-varying optima under strong convexity and smoothness.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

1d ago

Adaptive Optimization via Momentum on Variance-Normalized Gradients

arXiv:2602. 10204v2 Announce Type: replace Abstract: We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization.

By Francisco Patitucci, Aryan Mokhtari

llms

arXiv Machine Learning

Jul 10

Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

arXiv:2607. 08104v1 Announce Type: new Abstract: Stochastic gradient descent (SGD) is a cornerstone of modern optimization.

By Ryusei Yamada, Naoki Sato, Hideaki Iiduka

arXiv Machine Learning

Jun 11

Momentum LMS Theory beyond Stationarity: Stability, Tracking, and Regret

arXiv:2602. 11995v2 Announce Type: replace Abstract: In large-scale data processing scenarios, data often arrive in sequential streams generated by complex systems that exhibit drifting distributions and time-varying system parameters.

By Yifei Jin, Xin Zheng, Lei Guo

arXiv Machine Learning

Jun 2

Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients

arXiv:2512. 02342v3 Announce Type: replace-cross Abstract: The stochastic Polyak step size (SPS) has proven to be a promising choice for stochastic gradient descent (SGD), delivering competitive performance relative to state-of-the-art methods on smooth convex and non-convex optimization problems, including deep neural network training.

By Dimitris Oikonomou, Nicolas Loizou

benchmarks

arXiv Machine Learning

Jun 16

OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

arXiv:2603. 09923v4 Announce Type: replace Abstract: Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam.

By Ganzhao Yuan

arXiv Machine Learning

5d ago

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

arXiv:2608. 12925v1 Announce Type: new Abstract: Momentum-based optimizers are widely used in modern deep learning, yet the relations among momentum recursion, update geometry, and acceleration remain only partially understood.

By Zhixin Ren, Yau Lyu, Congrong Li, Liping Zhang, Shengbo Eben Li

reinforcement-learning

Related stories

Adaptive Optimization via Momentum on Variance-Normalized Gradients

Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

Momentum LMS Theory beyond Stationarity: Stability, Tracking, and Regret

Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients

OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization