arXiv Machine Learning
Weak-$L^\alpha$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 30
High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness
arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...
arXiv Machine Learning
Sep 14
Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H\"{o}lder Smoothness
arXiv:2609. 12785v1 Announce Type: new Abstract: Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice.
arXiv Machine Learning
Oct 16
On the Last-Iterate Convergence of Clipped Gradient Methods
The paper establishes the first optimal last‑iterate convergence rates, in high probability, for Clipped‑SGD when the gradient noise is heavy‑tailed. It addresses the gap left by earlier results that required strong noise assumptions, such as exponentially decaying tails. By proving these rates, the authors provide theoretical support for the practical effectiveness of gradient clipping in stochastic convex optimization.
arXiv Machine Learning
Sep 14
Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise
The paper proves that stochastic gradient descent with gradient clipping and additive Gaussian noise (SGD‑CN) converges almost surely under smoothness and bounded noise assumptions, given standard decaying step sizes. The analysis extends to momentum variants such as the stochastic heavy ball and Nesterov's accelerated gradient, showing that careful energy constructions yield similar guarantees. These results provide stronger theoretical foundations for understanding the pathwise behaviour of clipped stochastic gradient methods in both convex and nonconvex regimes.
arXiv Machine Learning
Sep 29
Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothness
arXiv:2609. 30276v1 Announce Type: new Abstract: We analyze the original same-step coordinate-wise AdaGrad under generalized smoothness and heavy-tailed noise with bounded variance.
arXiv Machine Learning
Jun 4
Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization
arXiv:2602. 05657v2 Announce Type: replace Abstract: The study of tail behaviour of SGD-induced processes has been attracting a lot of interest, due to offering strong guarantees with respect to individual runs of an algorithm.