arXiv Machine Learning September 14, 2026 By Amartya Mukherjee, Jun Liu

Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise

Read the original on arXiv Machine Learning →

The paper proves that stochastic gradient descent with gradient clipping and additive Gaussian noise (SGD‑CN) converges almost surely under smoothness and bounded noise assumptions, given standard decaying step sizes. The analysis extends to momentum variants such as the stochastic heavy ball and Nesterov's accelerated gradient, showing that careful energy constructions yield similar guarantees. These results provide stronger theoretical foundations for understanding the pathwise behaviour of clipped stochastic gradient methods in both convex and nonconvex regimes.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

safety

arXiv Machine Learning

Sep 30

High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness

arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...

By Taha El Bakkali El Kadi, Savelii Chezhegov, Aleksandr Beznosikov, Samuel Horv\'ath, Eduard Gorbunov

safety

arXiv Machine Learning

Jul 10

Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

arXiv:2607. 08104v1 Announce Type: new Abstract: Stochastic gradient descent (SGD) is a cornerstone of modern optimization.

By Ryusei Yamada, Naoki Sato, Hideaki Iiduka

arXiv Machine Learning

Oct 16

On the Last-Iterate Convergence of Clipped Gradient Methods

The paper establishes the first optimal last‑iterate convergence rates, in high probability, for Clipped‑SGD when the gradient noise is heavy‑tailed. It addresses the gap left by earlier results that required strong noise assumptions, such as exponentially decaying tails. By proving these rates, the authors provide theoretical support for the practical effectiveness of gradient clipping in stochastic convex optimization.

By Zijian Liu

arXiv Machine Learning

Oct 21

Weak-$L^\alpha$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

arXiv:2610.28107v1 Announce Type: cross Abstract: Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise. Ex...

By Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov, Alexander Gasnikov, Margarita Preobrazhenskaia, Igor Preobrazhensky, Eduard Gorbunov

safety

arXiv Machine Learning

Jun 2

In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise

arXiv:2606. 00520v1 Announce Type: cross Abstract: Many stochastic gradient methods are believed not to converge when the noise in stochastic gradients has only a finite $p$-th moment for $p\in\left(1,2\right)$, a setting known as the heavy-tailed noise assumption.

By Zijian Liu

arXiv AI

Sep 10

The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization

arXiv:2609.06064v1 Announce Type: cross Abstract: Stochastic min-max optimization has attracted increasing attention due to its applications in modern machine learning, while existing theoretical stu...

By Tianxi Zhu, Yi Xu, Xiangyang Ji

One email a morning, machine-written

Related stories

High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness

Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

On the Last-Iterate Convergence of Clipped Gradient Methods

Weak-$L^\alpha$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise

The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization