arXiv Machine Learning
Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 5
Gradient Descent's Last Iterate is Often (slightly) Suboptimal
arXiv:2604.13870v2 Announce Type: replace-cross Abstract: We consider the well-studied setting of minimizing a convex Lipschitz function using either gradient descent (GD) or its stochastic variant (...
arXiv Machine Learning
Sep 30
High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness
arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...
arXiv Machine Learning
Aug 12
A lower bound for stepsize-based acceleration of gradient descent
arXiv:2608. 10418v1 Announce Type: cross Abstract: Recent work has shown that, for smooth convex optimization, plain gradient descent can be accelerated from its textbook convergence rate of $O(T^{-1})$ (where $T$ denotes the number of iterations) to $O\big(T^{-\log_2(1+\sqrt{2})}\big)$ using carefully designed stepsize schedules alone, without resorting to momentum or other algorithmic modifications.
arXiv Machine Learning
Aug 12
Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization
arXiv:2310. 15976v4 Announce Type: replace Abstract: signSGD is attractive in nonconvex optimization because it communicates sign-valued rather than full-precision gradients.
arXiv Machine Learning
5d ago
Generalization Guarantees of Randomized Block Coordinate Descent for Nonconvex Learning
The paper presents high‑probability generalization guarantees for Randomized Block Coordinate Descent (RBCD) when applied to nonconvex learning problems. It derives bounds on the gradients of both empirical and population risks under standard smoothness assumptions, and shows how polynomially decaying step sizes and early stopping can achieve optimal convergence rates for population gradients. In settings where the objective is gradient‑dominated, the authors provide functional suboptimality bounds that approach optimistic rates seen in convex or low‑noise scenarios, and they also establish dimension‑independent bounds for loss functions with a particular structure, extending applicability to high‑dimensional problems.
arXiv Machine Learning
5d ago
On the Last-Iterate Convergence of Clipped Gradient Methods
The paper establishes the first optimal last‑iterate convergence rates, in high probability, for Clipped‑SGD when the gradient noise is heavy‑tailed. It addresses the gap left by earlier results that required strong noise assumptions, such as exponentially decaying tails. By proving these rates, the authors provide theoretical support for the practical effectiveness of gradient clipping in stochastic convex optimization.