arXiv Machine Learning July 31, 2026 By Yuichi Kondo, Hideaki Iiduka

Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis

Read the original on arXiv Machine Learning →

arXiv:2508. 03105v3 Announce Type: replace Abstract: We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 31

Towards joint scaling laws with optimal batch size schedules

arXiv:2607. 27731v1 Announce Type: new Abstract: Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics.

By Jiaxiang Li, Zhiqi Bu, Shiyun Xu

llms

arXiv Machine Learning

Jul 1

SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates

arXiv:2606. 30930v1 Announce Type: cross Abstract: Modern deep learning has been shown to operate at the edge of stability, routinely using learning rates far larger than those justified by classical optimization theory.

By Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun, Rene Vidal

arXiv Machine Learning

Jun 17

The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient

arXiv:2602. 11557v2 Announce Type: replace Abstract: A variety of widely used optimization methods like SignSGD and Muon can be interpreted as instances of steepest descent under different norm-induced geometries.

By Jichu Li, Xuan Tang, Difan Zou

safety

arXiv Machine Learning

Jul 7

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).

By Steffen Dereich, Arnulf Jentzen, Adrian Riekert

arXiv Machine Learning

1d ago

Scaling Laws for Dynamic Mini-Batch SGD in Sketched Linear Regression

arXiv:2605. 24316v3 Announce Type: replace Abstract: Mini-batching is central to large-scale optimization, yet its role in statistical scaling laws remains limited.

By Ziyan Chen, Zhongzhu Zhou, Ding-Xuan Zhou

safety

arXiv Machine Learning

Jul 30

Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent

arXiv:2606. 06772v2 Announce Type: replace-cross Abstract: Characterizing the optimization dynamics and statistical performance of over-parameterized deep neural networks (DNNs) remains a central challenge in understanding the remarkable success of deep learning.

By Junyu Zhou, Puyu Wang, Dennis Wagner, Yunwen Lei, Marius Kloft, Yiming Ying

Related stories

Towards joint scaling laws with optimal batch size schedules

SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates

The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

Scaling Laws for Dynamic Mini-Batch SGD in Sketched Linear Regression

Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent