arXiv AI

Second-Moment Stochastic Approximation Methods

arXiv Machine Learning

Sep 2

Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

The paper establishes uniform a priori bounds for the Adam optimizer, enabling an unconditional error analysis for a broad class of strongly convex stochastic optimization problems. Prior analyses were conditional, assuming Adam remained bounded, whereas this work removes that assumption. The results provide a rigorous foundation for Adam’s performance in training deep neural networks and other convex optimization tasks.

By Steffen Dereich, Thang Do, Arnulf Jentzen

arXiv Machine Learning

Aug 6

On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations

arXiv:2608. 04607v1 Announce Type: cross Abstract: Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs).

By Thang Do, Steffen Dereich, Arnulf Jentzen

llms

arXiv Statistics ML

6d ago

Distribution of hitting times for dissipative random dynamical systems on $\mathbb{R}^d$, with application to stochastic gradient descent

arXiv:2609.30274v1 Announce Type: new Abstract: Machine Learning and more specifically Deep Learning involves solving large scale nonconvex optimization problems. Several algorithms have been propose...

By St\'ephane Galatolo, St\'ephane Chr\'etien

arXiv Machine Learning

Jun 30

Analysis of Adam Algorithms for Stochastic Dynamic Systems

arXiv:2606. 28879v1 Announce Type: new Abstract: The adaptive moment estimation algorithm, known as Adam, is widely used in modern machine learning, owing to its low per-iteration complexity and strong empirical performance.

By Xin Zheng, Yifei Jin, Lei Guo

arXiv Machine Learning

Aug 18

On the Principles Behind Neural Network Optimizers

arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.

By Yushun Zhang

llms efficiency

arXiv Machine Learning

Jul 31

The Convergence Behavior of Adam under Heavy-Tailed Noise

arXiv:2607. 27383v1 Announce Type: new Abstract: We establish the first convergence guarantees for the plain vector-form \emph{Adam} optimizer under heavy-tailed stochastic noise.

By Yijiang Pang

arXiv Machine Learning

Sep 1

Convergence rates for the RMSprop optimizer with full control of the hyperparameters

arXiv:2608.30382v1 Announce Type: new Abstract: Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW o...

By Steffen Dereich, Arnulf Jentzen

arXiv AI

4d ago

Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients

arXiv:2609.37787v1 Announce Type: new Abstract: Adam is widely observed to remain stable even when the objective deviates significantly from global smoothness. Under the generalized smoothness framew...

By Ruinan Jin, Difei Cheng, Ling Chen, Jun Luo, Hao Zhou, Youzhi Zhang

arXiv Statistics ML

Aug 26

Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise

arXiv:2506.04192v4 Announce Type: replace-cross Abstract: Stochastic Frank-Wolfe is a classical optimization method for solving constrained optimization problems. On the other hand, recent optimizers...

By Maria-Eleni Sfyraki, Jun-Kun Wang

arXiv Machine Learning

Jul 7

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).

By Steffen Dereich, Arnulf Jentzen, Adrian Riekert

arXiv Machine Learning

Jul 7

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

arXiv:2607. 04233v1 Announce Type: cross Abstract: Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems.

By Shokhrukh Ibragimov, Arnulf Jentzen

arXiv Machine Learning

Jun 2

In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise

arXiv:2606. 00520v1 Announce Type: cross Abstract: Many stochastic gradient methods are believed not to converge when the noise in stochastic gradients has only a finite $p$-th moment for $p\in\left(1,2\right)$, a setting known as the heavy-tailed noise assumption.

By Zijian Liu

Related stories