arXiv:2610.28561v1 Announce Type: new
Abstract: Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in no...
By Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut, Ulrich Terstiege
arXiv:2610.19444v1 Announce Type: cross
Abstract: Random reshuffling stochastic gradient descent (RR-SGD) is widely used for finite-sum optimization in machine learning. For convex $L$-smooth compone...
By Zijun Chen, Zihan Zhang
arXiv:2505.20817v3 Announce Type: replace-cross
Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...
By Taha El Bakkali El Kadi, Savelii Chezhegov, Aleksandr Beznosikov, Samuel Horv\'ath, Eduard Gorbunov
The paper presents high‑probability generalization guarantees for Randomized Block Coordinate Descent (RBCD) when applied to nonconvex learning problems. It derives bounds on the gradients of both empirical and population risks under standard smoothness assumptions, and shows how polynomially decaying step sizes and early stopping can achieve optimal convergence rates for population gradients. In settings where the objective is gradient‑dominated, the authors provide functional suboptimality bounds that approach optimistic rates seen in convex or low‑noise scenarios, and they also establish dimension‑independent bounds for loss functions with a particular structure, extending applicability to high‑dimensional problems.
By Ruixi Hu, Liang Wu
arXiv:2512. 02342v3 Announce Type: replace-cross Abstract: The stochastic Polyak step size (SPS) has proven to be a promising choice for stochastic gradient descent (SGD), delivering competitive performance relative to state-of-the-art methods on smooth convex and non-convex optimization problems, including deep neural network training.
By Dimitris Oikonomou, Nicolas Loizou
arXiv:2610. 42048v1 Announce Type: cross Abstract: This work considers the design of stepsize schedules for stochastic gradient descent in smooth convex optimization.
By Benjamin Grimmer, Qining Shen
The paper proves that stochastic gradient descent with gradient clipping and additive Gaussian noise (SGD‑CN) converges almost surely under smoothness and bounded noise assumptions, given standard decaying step sizes. The analysis extends to momentum variants such as the stochastic heavy ball and Nesterov's accelerated gradient, showing that careful energy constructions yield similar guarantees. These results provide stronger theoretical foundations for understanding the pathwise behaviour of clipped stochastic gradient methods in both convex and nonconvex regimes.
By Amartya Mukherjee, Jun Liu
arXiv:2609.08380v1 Announce Type: cross
Abstract: We study the stochastic first-order oracle complexity for constrained or regularized convex-concave min-max optimization and stochastic monotone vari...
By Ahmet Alacaoglu
arXiv:2508.04860v2 Announce Type: replace-cross
Abstract: Stochastic Gradient Descent (SGD) is a cornerstone of large-scale optimization, yet its theoretical behavior under heavy-tailed noise -- comm...
By Ilyas Fatkhullin, Florian H\"ubler, Guanghui Lan
The paper studies a variant of stochastic gradient descent called SGDIR, which incorporates initial regularization. It derives dimension‑free upper bounds on the expected excess risk for the squared loss, providing new rates for both averaged and non‑averaged SGDIR under various assumptions. The authors also establish matching lower bounds in certain regimes and compare SGDIR to ridge regression in noisy settings, showing comparable performance up to a polylogarithmic factor.
By Nabil Kahal\'e
arXiv:2608. 25551v1 Announce Type: new Abstract: Stochastic gradient descent (SGD) is typically analyzed at a deterministic horizon chosen before the algorithm is run, even though practical stopping decisions are made adaptively by inspecting the evolving trajectory.
By Liviu Aolaritei, Lucas L\'evy, Francis Bach, Michael I. Jordan
arXiv:2406. 13041v3 Announce Type: replace Abstract: Lower-bound analyses for nonconvex strongly-concave minimax optimization problems have shown that stochastic first-order algorithms require at least $\mathcal{O}(\varepsilon^{-4})$ sample complexity to find an $\varepsilon$-stationary point.
By Haoyuan Cai, Sulaiman A. Alghunaim, Ali H. Sayed