High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2610.28107v1 Announce Type: cross Abstract: Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise. Ex...
The paper proves that stochastic gradient descent with gradient clipping and additive Gaussian noise (SGD‑CN) converges almost surely under smoothness and bounded noise assumptions, given standard decaying step sizes. The analysis extends to momentum variants such as the stochastic heavy ball and Nesterov's accelerated gradient, showing that careful energy constructions yield similar guarantees. These results provide stronger theoretical foundations for understanding the pathwise behaviour of clipped stochastic gradient methods in both convex and nonconvex regimes.
The paper establishes the first optimal last‑iterate convergence rates, in high probability, for Clipped‑SGD when the gradient noise is heavy‑tailed. It addresses the gap left by earlier results that required strong noise assumptions, such as exponentially decaying tails. By proving these rates, the authors provide theoretical support for the practical effectiveness of gradient clipping in stochastic convex optimization.
arXiv:2609. 12785v1 Announce Type: new Abstract: Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice.
arXiv:2609.06064v1 Announce Type: cross Abstract: Stochastic min-max optimization has attracted increasing attention due to its applications in modern machine learning, while existing theoretical stu...
arXiv:2602. 05657v2 Announce Type: replace Abstract: The study of tail behaviour of SGD-induced processes has been attracting a lot of interest, due to offering strong guarantees with respect to individual runs of an algorithm.