arXiv Machine Learning
Gradient Descent's Last Iterate is Often (slightly) Suboptimal
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 12
A lower bound for stepsize-based acceleration of gradient descent
arXiv:2608. 10418v1 Announce Type: cross Abstract: Recent work has shown that, for smooth convex optimization, plain gradient descent can be accelerated from its textbook convergence rate of $O(T^{-1})$ (where $T$ denotes the number of iterations) to $O\big(T^{-\log_2(1+\sqrt{2})}\big)$ using carefully designed stepsize schedules alone, without resorting to momentum or other algorithmic modifications.
arXiv Machine Learning
6d ago
Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
arXiv:2610.19444v1 Announce Type: cross Abstract: Random reshuffling stochastic gradient descent (RR-SGD) is widely used for finite-sum optimization in machine learning. For convex $L$-smooth compone...
arXiv Machine Learning
5d ago
On the Last-Iterate Convergence of Clipped Gradient Methods
The paper establishes the first optimal last‑iterate convergence rates, in high probability, for Clipped‑SGD when the gradient noise is heavy‑tailed. It addresses the gap left by earlier results that required strong noise assumptions, such as exponentially decaying tails. By proving these rates, the authors provide theoretical support for the practical effectiveness of gradient clipping in stochastic convex optimization.
arXiv Machine Learning
Sep 10
The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives
arXiv:2609.08537v1 Announce Type: cross Abstract: We study the time-uniform convergence of the raw iterate of standard stochastic gradient descent (SGD) for unconstrained smooth convex objectives. We...
arXiv Machine Learning
Jun 24
New Bounds for the Last Iterate of the Stochastic subGradient Method
arXiv:2606. 24879v1 Announce Type: cross Abstract: We study the last iterate of the stochastic subgradient method for one-dimensional convex Lipschitz objectives.
arXiv Machine Learning
Jun 2
Accelerating Min-Max Optimization via Power-Law Stepsizes
arXiv:2606. 01764v1 Announce Type: cross Abstract: We revisit the convergence guarantees of the Extragradient (EG) method for unconstrained biaffine min-max optimization.