arXiv Machine Learning
Bilevel Learning via Strategically Inexact Stochastic Gradient Descent
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 21
Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training
arXiv:2608.09523v2 Announce Type: replace Abstract: Stochastic gradient descent (SGD) and its variants train deep neural networks (DNNs) effectively in practice, yet classical optimization theory doe...
arXiv Machine Learning
Jul 7
Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).
arXiv Machine Learning
Oct 7
A Single-Loop, Constant-Batch First-Order Penalty Method for Stochastic Bilevel Optimization
arXiv:2610.07290v1 Announce Type: cross Abstract: Recent advances in penalty-based methods for stochastic bilevel optimization (SBO) have eliminated the need for second-order derivative oracles. Howe...
arXiv Machine Learning
Sep 29
To Solve Bilevel Optimization with Nonconvex Lower Levels, We Need Second-Order Stationarity
arXiv:2609. 30501v1 Announce Type: new Abstract: Although bilevel optimization (BLO) has emerged as a powerful framework for addressing many complex and nested machine learning problems in recent years, most existing studies are confined to the lower-level strongly convex (LLSC) or lower-level generally convex (LLGC) settings (i.
arXiv Machine Learning
Aug 26
AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
arXiv:2402.11215v4 Announce Type: replace Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-sc...
arXiv Machine Learning
Jul 15
Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps
arXiv:2607. 12360v1 Announce Type: new Abstract: The cooldown phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining, lowers the final training loss in some settings and does nothing in others.