arXiv Machine Learning September 14, 2026 By Amartya Mukherjee, Jun Liu

Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise

Read the original on arXiv Machine Learning →

The paper proves that stochastic gradient descent with gradient clipping and additive Gaussian noise (SGD‑CN) converges almost surely under smoothness and bounded noise assumptions, given standard decaying step sizes. The analysis extends to momentum variants such as the stochastic heavy ball and Nesterov's accelerated gradient, showing that careful energy constructions yield similar guarantees. These results provide stronger theoretical foundations for understanding the pathwise behaviour of clipped stochastic gradient methods in both convex and nonconvex regimes.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

safety

arXiv Machine Learning

Jul 10

Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

arXiv:2607. 08104v1 Announce Type: new Abstract: Stochastic gradient descent (SGD) is a cornerstone of modern optimization.

By Ryusei Yamada, Naoki Sato, Hideaki Iiduka

arXiv Machine Learning

Jun 2

In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise

arXiv:2606. 00520v1 Announce Type: cross Abstract: Many stochastic gradient methods are believed not to converge when the noise in stochastic gradients has only a finite $p$-th moment for $p\in\left(1,2\right)$, a setting known as the heavy-tailed noise assumption.

By Zijian Liu

arXiv AI

Sep 10

The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization

arXiv:2609.06064v1 Announce Type: cross Abstract: Stochastic min-max optimization has attracted increasing attention due to its applications in modern machine learning, while existing theoretical stu...

By Tianxi Zhu, Yi Xu, Xiangyang Ji

arXiv Machine Learning

3d ago

Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H\"{o}lder Smoothness

arXiv:2609. 12785v1 Announce Type: new Abstract: Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice.

By Misbah Uz Zaman, Anirbit Mukherjee

arXiv Machine Learning

Jun 24

Robust and Fast Training via Per-Sample Clipping

arXiv:2605. 02701v2 Announce Type: replace-cross Abstract: We propose a robust gradient estimator based on per-sample gradient clipping and analyze its properties both theoretically and empirically.

By Davide Nobile, Philipp Grohs

arXiv Machine Learning

Jun 2

Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients

arXiv:2512. 02342v3 Announce Type: replace-cross Abstract: The stochastic Polyak step size (SPS) has proven to be a promising choice for stochastic gradient descent (SGD), delivering competitive performance relative to state-of-the-art methods on smooth convex and non-convex optimization problems, including deep neural network training.

By Dimitris Oikonomou, Nicolas Loizou

benchmarks

One email a morning, machine-written

Related stories

Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise

The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization

Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H\"{o}lder Smoothness

Robust and Fast Training via Per-Sample Clipping

Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients