arXiv:2605. 18694v2 Announce Type: replace-cross Abstract: Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process.
arXiv:2602. 05657v2 Announce Type: replace Abstract: The study of tail behaviour of SGD-induced processes has been attracting a lot of interest, due to offering strong guarantees with respect to individual runs of an algorithm.
The paper establishes uniform a priori bounds for the Adam optimizer, enabling an unconditional error analysis for a broad class of strongly convex stochastic optimization problems. Prior analyses were conditional, assuming Adam remained bounded, whereas this work removes that assumption. The results provide a rigorous foundation for Adam’s performance in training deep neural networks and other convex optimization tasks.
arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...
arXiv:2607. 04233v1 Announce Type: cross Abstract: Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems.
arXiv:2310. 15976v4 Announce Type: replace Abstract: signSGD is attractive in nonconvex optimization because it communicates sign-valued rather than full-precision gradients.
arXiv:2606. 28879v1 Announce Type: new Abstract: The adaptive moment estimation algorithm, known as Adam, is widely used in modern machine learning, owing to its low per-iteration complexity and strong empirical performance.
arXiv:2603. 09923v4 Announce Type: replace Abstract: Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam.
The paper introduces Curvature-Scaled Sharpness-Aware Minimization (CS‑SAM), a variant of SAM that scales both the perturbation and descent step by curvature bounds to uniformly control gradient distortion under generalized smoothness conditions. It proves deterministic ≥1/T convergence rates for nonconvex objectives, last‑iterate ≥1/T rates for convex objectives, and linear convergence under the Polyak‑Łojasiewicz condition, while recovering standard gradient descent rates when the perturbation is zero. Experiments on computer vision tasks show that CS‑SAM, particularly the (H0,H1) variant, achieves the highest mean best‑test accuracy across four settings compared to tuned SAM and USAM.
arXiv:2508.04860v2 Announce Type: replace-cross Abstract: Stochastic Gradient Descent (SGD) is a cornerstone of large-scale optimization, yet its theoretical behavior under heavy-tailed noise -- comm...
arXiv:2609.37425v1 Announce Type: cross Abstract: Fix a target accuracy $\varepsilon$, a gradient-noise level $s$, and a horizon $N$. We wish to design algorithms which minimize the probability of ob...
arXiv:2601. 07326v4 Announce Type: replace-cross Abstract: This paper studies AdamW-style Shampoo, an effective variant of the classical Shampoo that won the external tuning track of the AlgoPerf neural network training competition.