arXiv Machine Learning June 18, 2026 By Jean-Fran\c{c}ois Aujol, J\'er\'emie Bigot, Camille Castera

Stochastic Adaptive Gradient Descent Without Descent

Read the original on arXiv Machine Learning →

arXiv:2509. 14969v2 Announce Type: replace Abstract: We introduce a new adaptive step-size strategy for convex optimization with stochastic gradient that exploits the local geometry of the objective function only by means of a first-order stochastic oracle and without any hyper-parameter tuning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 2

Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients

arXiv:2512. 02342v3 Announce Type: replace-cross Abstract: The stochastic Polyak step size (SPS) has proven to be a promising choice for stochastic gradient descent (SGD), delivering competitive performance relative to state-of-the-art methods on smooth convex and non-convex optimization problems, including deep neural network training.

By Dimitris Oikonomou, Nicolas Loizou

benchmarks

arXiv Machine Learning

Jun 2

Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler

arXiv:2606. 01827v1 Announce Type: cross Abstract: Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted optimizer for training machine learning models.

By Dimitris Oikonomou, Nicolas Loizou

arXiv Machine Learning

Aug 26

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods

arXiv:2402.11215v4 Announce Type: replace Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-sc...

By Tim Tsz-Kit Lau, Han Liu, Mladen Kolar

arXiv Machine Learning

Jun 2

Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad

arXiv:2605. 18694v2 Announce Type: replace-cross Abstract: Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process.

By Zijian Liu

arXiv Machine Learning

Jul 28

Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided H\"older Regularity

arXiv:2607. 22906v1 Announce Type: new Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided H\"older regularity.

By Arzu Ahmadova, Ismail Huseynov

benchmarks

arXiv Machine Learning

Sep 16

Federated stochastic bilevel optimization with fully first-order gradients

arXiv:2609.16350v1 Announce Type: new Abstract: Federated stochastic bilevel optimization has been actively studied in recent years due to its widespread applications in machine learning. However, mo...

By Yihan Zhang, Rohit Dhaipule, Chiu C Tan, Haibin Ling, Hongchang Gao

One email a morning, machine-written

Related stories

Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients

Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods

Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad

Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided H\"older Regularity

Federated stochastic bilevel optimization with fully first-order gradients