arXiv Machine Learning September 14, 2026 By Misbah Uz Zaman, Anirbit Mukherjee

Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H\"{o}lder Smoothness

Read the original on arXiv Machine Learning →

arXiv:2609. 12785v1 Announce Type: new Abstract: Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Oct 21

Weak-$L^\alpha$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

arXiv:2610.28107v1 Announce Type: cross Abstract: Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise. Ex...

By Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov, Alexander Gasnikov, Margarita Preobrazhenskaia, Igor Preobrazhensky, Eduard Gorbunov

safety

arXiv Machine Learning

Sep 30

High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness

arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...

By Taha El Bakkali El Kadi, Savelii Chezhegov, Aleksandr Beznosikov, Samuel Horv\'ath, Eduard Gorbunov

safety

arXiv Machine Learning

Oct 16

On the Last-Iterate Convergence of Clipped Gradient Methods

The paper establishes the first optimal last‑iterate convergence rates, in high probability, for Clipped‑SGD when the gradient noise is heavy‑tailed. It addresses the gap left by earlier results that required strong noise assumptions, such as exponentially decaying tails. By proving these rates, the authors provide theoretical support for the practical effectiveness of gradient clipping in stochastic convex optimization.

By Zijian Liu

arXiv Machine Learning

Sep 25

Precise Convergence Speed of Clipped SGD

arXiv:2609. 29458v1 Announce Type: new Abstract: We present a tightened convergence analysis of clipped gradient descent on $(L_0, L_1)$-smooth functions, with quantitative constants.

By David A. R. Robin

safety

arXiv Machine Learning

Jun 4

Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization

arXiv:2602. 05657v2 Announce Type: replace Abstract: The study of tail behaviour of SGD-induced processes has been attracting a lot of interest, due to offering strong guarantees with respect to individual runs of an algorithm.

By Aleksandar Armacki, Dragana Bajovi\'c, Du\v{s}an Jakoveti\'c, Soummya Kar, Ali H. Sayed

arXiv Machine Learning

Jun 2

Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad

arXiv:2605. 18694v2 Announce Type: replace-cross Abstract: Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process.

By Zijian Liu

One email a morning, machine-written

Related stories

Weak-$L^\alpha$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness

On the Last-Iterate Convergence of Clipped Gradient Methods

Precise Convergence Speed of Clipped SGD

Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization

Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad