← Back to all news
arXiv Machine Learning By Yuanzhe Tao, Yifeng Liu, Huizhuo Yuan, Xun Zhou, Yuan Cao, Quanquan Gu

Towards Simple and Provable Parameter-Free Adaptive Gradient Methods

Read the original on arXiv Machine Learning →

arXiv:2412. 19444v2 Announce Type: replace Abstract: Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jul 31

Towards Stability of Parameter-Free Optimization

arXiv:2405. 04376v4 Announce Type: replace Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge.

By Yijiang Pang, Shuyang Yu, Bao Hoang, Jiayu Zhou
More like this →
arXiv Machine Learning
Jul 7

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).

By Steffen Dereich, Arnulf Jentzen, Adrian Riekert
More like this →
arXiv Machine Learning
Sep 11

ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks

ExpTest is an autonomous learning‑rate controller that uses the training loss curve as an online signal to perform sequential statistical tests on theoretically motivated windows, detecting convergent behavior and triggering learning‑rate reductions. It combines a covariance‑based initial learning‑rate estimate, curvature‑motivated window sizing, and a two‑phase test‑driven decay, relying on the approximately exponential decay predicted under linearized network dynamics. Experiments on regression, classification, forecasting, and natural‑language tasks across various architectures show that ExpTest achieves competitive performance compared to hand‑tuned SGD baselines and recent learning‑rate‑free methods, without requiring manual initial learning‑rate selection or predefined scheduling.

By Zan Chaudhry, Naoko Mizuno
llmsagents
More like this →
arXiv Machine Learning
Jul 30

Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent

arXiv:2606. 06772v2 Announce Type: replace-cross Abstract: Characterizing the optimization dynamics and statistical performance of over-parameterized deep neural networks (DNNs) remains a central challenge in understanding the remarkable success of deep learning.

By Junyu Zhou, Puyu Wang, Dennis Wagner, Yunwen Lei, Marius Kloft, Yiming Ying
More like this →