arXiv:2610.28107v1 Announce Type: cross
Abstract: Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise. Ex...
By Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov, Alexander Gasnikov, Margarita Preobrazhenskaia, Igor Preobrazhensky, Eduard Gorbunov
The paper proves that stochastic gradient descent with gradient clipping and additive Gaussian noise (SGD‑CN) converges almost surely under smoothness and bounded noise assumptions, given standard decaying step sizes. The analysis extends to momentum variants such as the stochastic heavy ball and Nesterov's accelerated gradient, showing that careful energy constructions yield similar guarantees. These results provide stronger theoretical foundations for understanding the pathwise behaviour of clipped stochastic gradient methods in both convex and nonconvex regimes.
By Amartya Mukherjee, Jun Liu
The paper establishes the first optimal last‑iterate convergence rates, in high probability, for Clipped‑SGD when the gradient noise is heavy‑tailed. It addresses the gap left by earlier results that required strong noise assumptions, such as exponentially decaying tails. By proving these rates, the authors provide theoretical support for the practical effectiveness of gradient clipping in stochastic convex optimization.
By Zijian Liu
arXiv:2609. 12785v1 Announce Type: new Abstract: Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice.
By Misbah Uz Zaman, Anirbit Mukherjee
arXiv:2609.06064v1 Announce Type: cross
Abstract: Stochastic min-max optimization has attracted increasing attention due to its applications in modern machine learning, while existing theoretical stu...
By Tianxi Zhu, Yi Xu, Xiangyang Ji
arXiv:2602. 05657v2 Announce Type: replace Abstract: The study of tail behaviour of SGD-induced processes has been attracting a lot of interest, due to offering strong guarantees with respect to individual runs of an algorithm.
By Aleksandar Armacki, Dragana Bajovi\'c, Du\v{s}an Jakoveti\'c, Soummya Kar, Ali H. Sayed
arXiv:2508.04860v2 Announce Type: replace-cross
Abstract: Stochastic Gradient Descent (SGD) is a cornerstone of large-scale optimization, yet its theoretical behavior under heavy-tailed noise -- comm...
By Ilyas Fatkhullin, Florian H\"ubler, Guanghui Lan
The paper presents a non‑asymptotic analysis of Stochastic Gradient Descent (SGD) that focuses on the bias component of its convergence bounds. Under assumptions of convexity or strong convexity and smoothness, the authors derive new bias‑optimal bounds for all constant step‑sizes <gamma L> in (0,2), covering critical and large step‑size regimes that were previously unexplored without extra variance assumptions. For critical step‑sizes, the paper shows that while an optimal bias term cannot be achieved, it can be approximated arbitrarily closely, and applies these results to obtain new complexity bounds for the Stochastic Proximal algorithm without requiring regularity assumptions.
By Daniel Cortild, Lucas Ketels, Juan Peypouquet, Guillaume Garrigos
arXiv:2611. 13846v1 Announce Type: cross Abstract: We establish last-iterate mean-square residual bounds for nonexpansive stochastic approximation (SA), or stochastic Krasnoselskii-Mann (KM) iterations.
By Ramanan Tamizholi, Siddharth Chandak, Rahul Singh
arXiv:2610.07290v1 Announce Type: cross
Abstract: Recent advances in penalty-based methods for stochastic bilevel optimization (SBO) have eliminated the need for second-order derivative oracles. Howe...
By Xingyu Chen, Ming Yang, Quanqi Hu, Tianbao Yang
arXiv:2608. 25551v1 Announce Type: new Abstract: Stochastic gradient descent (SGD) is typically analyzed at a deterministic horizon chosen before the algorithm is run, even though practical stopping decisions are made adaptively by inspecting the evolving trajectory.
By Liviu Aolaritei, Lucas L\'evy, Francis Bach, Michael I. Jordan
arXiv:2605. 18694v2 Announce Type: replace-cross Abstract: Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process.
By Zijian Liu