arXiv Machine Learning August 18, 2026 By Francisco Patitucci, Aryan Mokhtari

Adaptive Optimization via Momentum on Variance-Normalized Gradients

Read the original on arXiv Machine Learning →

arXiv:2602. 10204v2 Announce Type: replace Abstract: We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jul 10

Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

arXiv:2607. 08104v1 Announce Type: new Abstract: Stochastic gradient descent (SGD) is a cornerstone of modern optimization.

By Ryusei Yamada, Naoki Sato, Hideaki Iiduka

arXiv Machine Learning

Jun 17

MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization

arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.

By Da Chang, Ganzhao Yuan

llms fine-tuning safety

arXiv Machine Learning

Jun 9

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.

By Ganzhao Yuan

arXiv Machine Learning

Jul 31

The Convergence Behavior of Adam under Heavy-Tailed Noise

arXiv:2607. 27383v1 Announce Type: new Abstract: We establish the first convergence guarantees for the plain vector-form \emph{Adam} optimizer under heavy-tailed stochastic noise.

By Yijiang Pang

arXiv Machine Learning

Jun 16

OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

arXiv:2603. 09923v4 Announce Type: replace Abstract: Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam.

By Ganzhao Yuan

arXiv Machine Learning

5d ago

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

arXiv:2608. 12925v1 Announce Type: new Abstract: Momentum-based optimizers are widely used in modern deep learning, yet the relations among momentum recursion, update geometry, and acceleration remain only partially understood.

By Zhixin Ren, Yau Lyu, Congrong Li, Liping Zhang, Shengbo Eben Li

reinforcement-learning

Related stories

Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

The Convergence Behavior of Adam under Heavy-Tailed Noise

OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization