arXiv Machine Learning June 2, 2026 By Zijian Liu

Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad

Read the original on arXiv Machine Learning →

arXiv:2605. 18694v2 Announce Type: replace-cross Abstract: Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 2

In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise

arXiv:2606. 00520v1 Announce Type: cross Abstract: Many stochastic gradient methods are believed not to converge when the noise in stochastic gradients has only a finite $p$-th moment for $p\in\left(1,2\right)$, a setting known as the heavy-tailed noise assumption.

By Zijian Liu

arXiv Machine Learning

4d ago

High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness

arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...

By Taha El Bakkali El Kadi, Savelii Chezhegov, Aleksandr Beznosikov, Samuel Horv\'ath, Eduard Gorbunov

safety

arXiv Machine Learning

Sep 14

Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H\"{o}lder Smoothness

arXiv:2609. 12785v1 Announce Type: new Abstract: Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice.

By Misbah Uz Zaman, Anirbit Mukherjee

arXiv Machine Learning

Sep 1

Convergence rates for the RMSprop optimizer with full control of the hyperparameters

arXiv:2608.30382v1 Announce Type: new Abstract: Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW o...

By Steffen Dereich, Arnulf Jentzen

arXiv Machine Learning

Jul 31

The Convergence Behavior of Adam under Heavy-Tailed Noise

arXiv:2607. 27383v1 Announce Type: new Abstract: We establish the first convergence guarantees for the plain vector-form \emph{Adam} optimizer under heavy-tailed stochastic noise.

By Yijiang Pang

arXiv Machine Learning

5d ago

Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothness

arXiv:2609. 30276v1 Announce Type: new Abstract: We analyze the original same-step coordinate-wise AdaGrad under generalized smoothness and heavy-tailed noise with bounded variance.

By Alokendu Mazumder, Ayaan Mohd, Harshit Rawat, Arnab Roy, Mayank Baranwal, Punit Rathore

One email a morning, machine-written

Related stories

In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise

High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness

Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H\"{o}lder Smoothness

Convergence rates for the RMSprop optimizer with full control of the hyperparameters

The Convergence Behavior of Adam under Heavy-Tailed Noise

Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothness