arXiv AI
Second-Moment Stochastic Approximation Methods
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 2
Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method
The paper establishes uniform a priori bounds for the Adam optimizer, enabling an unconditional error analysis for a broad class of strongly convex stochastic optimization problems. Prior analyses were conditional, assuming Adam remained bounded, whereas this work removes that assumption. The results provide a rigorous foundation for Adam’s performance in training deep neural networks and other convex optimization tasks.
arXiv Machine Learning
Aug 6
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
arXiv:2608. 04607v1 Announce Type: cross Abstract: Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs).
arXiv Statistics ML
6d ago
Distribution of hitting times for dissipative random dynamical systems on $\mathbb{R}^d$, with application to stochastic gradient descent
arXiv:2609.30274v1 Announce Type: new Abstract: Machine Learning and more specifically Deep Learning involves solving large scale nonconvex optimization problems. Several algorithms have been propose...
arXiv Machine Learning
Jun 30
Analysis of Adam Algorithms for Stochastic Dynamic Systems
arXiv:2606. 28879v1 Announce Type: new Abstract: The adaptive moment estimation algorithm, known as Adam, is widely used in modern machine learning, owing to its low per-iteration complexity and strong empirical performance.
arXiv Machine Learning
Aug 18
On the Principles Behind Neural Network Optimizers
arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.
arXiv Machine Learning
Jul 31
The Convergence Behavior of Adam under Heavy-Tailed Noise
arXiv:2607. 27383v1 Announce Type: new Abstract: We establish the first convergence guarantees for the plain vector-form \emph{Adam} optimizer under heavy-tailed stochastic noise.