arXiv Machine Learning July 7, 2026 By Steffen Dereich, Arnulf Jentzen, Adrian Riekert

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

Read the original on arXiv Machine Learning →

arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 2

Towards Simple and Provable Parameter-Free Adaptive Gradient Methods

arXiv:2412. 19444v2 Announce Type: replace Abstract: Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process.

By Yuanzhe Tao, Yifeng Liu, Huizhuo Yuan, Xun Zhou, Yuan Cao, Quanquan Gu

arXiv AI

Jun 11

A Physics-Inspired Optimizer: Velocity Regularized Adam

arXiv:2505. 13196v3 Announce Type: replace-cross Abstract: We introduce Velocity-Regularized Adam (VRAdam), a physics-inspired optimizer for training deep neural networks that draws on ideas from quartic terms for kinetic energy with its stabilizing effects on various system dynamics.

By Pranav Vaidhyanathan, Lucas Schorling, Natalia Ares, Maike Osborne

llms computer-vision benchmarks

arXiv Machine Learning

Jul 2

Homogenization of $\ell_2$-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent

arXiv:2607. 00207v1 Announce Type: cross Abstract: We develop a framework for analyzing the learning dynamics of $\ell_2$-adversarial training of single-index models on Gaussian mixtures in the high-dimensional limit under streaming stochastic gradient descent (SGD).

By Fabrizzio Sabelli

safety

arXiv Machine Learning

Jul 30

Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent

arXiv:2606. 06772v2 Announce Type: replace-cross Abstract: Characterizing the optimization dynamics and statistical performance of over-parameterized deep neural networks (DNNs) remains a central challenge in understanding the remarkable success of deep learning.

By Junyu Zhou, Puyu Wang, Dennis Wagner, Yunwen Lei, Marius Kloft, Yiming Ying

arXiv Machine Learning

Aug 4

Simplified Quadratic Gradient: A Unified Framework Bridging Gradient Descent and Newton-Type Methods by Synthesizing Hessians and Gradients

arXiv:2209. 03282v5 Announce Type: replace-cross Abstract: Accelerating the convergence of second-order optimization, particularly Newton-type methods, remains a pivotal challenge in algorithmic research.

By John Chiang

arXiv Machine Learning

1d ago

On the Principles Behind Neural Network Optimizers

arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.

By Yushun Zhang

llms efficiency

Related stories

Towards Simple and Provable Parameter-Free Adaptive Gradient Methods

A Physics-Inspired Optimizer: Velocity Regularized Adam

Homogenization of $\ell_2$-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent

Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent

Simplified Quadratic Gradient: A Unified Framework Bridging Gradient Descent and Newton-Type Methods by Synthesizing Hessians and Gradients

On the Principles Behind Neural Network Optimizers