arXiv Statistics ML
A stochastic subgradient method with optimal failure exponent
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Jun 4
Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization
arXiv:2602. 05657v2 Announce Type: replace Abstract: The study of tail behaviour of SGD-induced processes has been attracting a lot of interest, due to offering strong guarantees with respect to individual runs of an algorithm.
arXiv Machine Learning
Aug 10
A proximal subgradient method for nonconvex stochastic optimization under the Kurdyka-{\L}ojasiewicz condition
arXiv:2608. 05460v1 Announce Type: cross Abstract: This work introduces a proximal stochastic subgradient method for minimizing the sum of an expected cost, whose integrand is potentially nonsmooth and nonconvex, and a lower semicontinuous, prox-bounded function.
arXiv Machine Learning
1d ago
High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness
arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...
arXiv Machine Learning
Jun 24
New Bounds for the Last Iterate of the Stochastic subGradient Method
arXiv:2606. 24879v1 Announce Type: cross Abstract: We study the last iterate of the stochastic subgradient method for one-dimensional convex Lipschitz objectives.
arXiv Machine Learning
Jun 2
Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad
arXiv:2605. 18694v2 Announce Type: replace-cross Abstract: Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process.
arXiv Machine Learning
Sep 1
Convergence rates for the RMSprop optimizer with full control of the hyperparameters
arXiv:2608.30382v1 Announce Type: new Abstract: Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW o...