arXiv Machine Learning

Bilevel Learning via Strategically Inexact Stochastic Gradient Descent

arXiv Machine Learning

Oct 21

Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training

arXiv:2608.09523v2 Announce Type: replace Abstract: Stochastic gradient descent (SGD) and its variants train deep neural networks (DNNs) effectively in practice, yet classical optimization theory doe...

By Binchuan Qi

llms

arXiv Machine Learning

Jul 7

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).

By Steffen Dereich, Arnulf Jentzen, Adrian Riekert

arXiv Machine Learning

Oct 7

A Single-Loop, Constant-Batch First-Order Penalty Method for Stochastic Bilevel Optimization

arXiv:2610.07290v1 Announce Type: cross Abstract: Recent advances in penalty-based methods for stochastic bilevel optimization (SBO) have eliminated the need for second-order derivative oracles. Howe...

By Xingyu Chen, Ming Yang, Quanqi Hu, Tianbao Yang

arXiv Machine Learning

Sep 29

To Solve Bilevel Optimization with Nonconvex Lower Levels, We Need Second-Order Stationarity

arXiv:2609. 30501v1 Announce Type: new Abstract: Although bilevel optimization (BLO) has emerged as a powerful framework for addressing many complex and nested machine learning problems in recent years, most existing studies are confined to the lower-level strongly convex (LLSC) or lower-level generally convex (LLGC) settings (i.

By Zhiyao Zhang, Menglu Yu, Alvaro Velasquez, Nathaniel D. Bastian, Jia Liu

llms benchmarks

arXiv Machine Learning

Aug 26

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods

arXiv:2402.11215v4 Announce Type: replace Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-sc...

By Tim Tsz-Kit Lau, Han Liu, Mladen Kolar

arXiv Machine Learning

Jul 15

Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

arXiv:2607. 12360v1 Announce Type: new Abstract: The cooldown phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining, lowers the final training loss in some settings and does nothing in others.

By Subham Singh, Ashutosh Mishra, Subha Raut

arXiv Machine Learning

Aug 11

Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.

By Binchuan Qi

safety

arXiv Machine Learning

Jun 16

A Fully First-Order Layer for Differentiable Optimization

arXiv:2512. 02494v2 Announce Type: replace Abstract: Differentiable optimization layers enable learning systems to make decisions by solving embedded optimization problems.

By Zihao Zhao, Kai-Chia Mo, Shing-Hei Ho, Brandon Amos, Kai Wang

arXiv AI

Jul 21

Reducing Per-Sample Harm in Stochastic Optimization

arXiv:2607. 16261v1 Announce Type: cross Abstract: Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.

By Apostolos Avranas

computer-vision benchmarks

arXiv Machine Learning

Aug 14

Efficient Hessian-Free Methods for Multi-Objective Bilevel Optimization with Nonconvex Lower Level

arXiv:2608. 12704v1 Announce Type: cross Abstract: Multi-objective bilevel optimization has wide applications in the AI area such as automated learning and multi-task meta-learning.

By Yicong Jiang, Feihu Huang

arXiv Machine Learning

Sep 16

Federated stochastic bilevel optimization with fully first-order gradients

arXiv:2609.16350v1 Announce Type: new Abstract: Federated stochastic bilevel optimization has been actively studied in recent years due to its widespread applications in machine learning. However, mo...

By Yihan Zhang, Rohit Dhaipule, Chiu C Tan, Haibin Ling, Hongchang Gao

arXiv Machine Learning

Aug 13

A New First-Order Meta-Learning Algorithm with Convergence Guarantees

arXiv:2409. 03682v2 Announce Type: replace Abstract: Learning new tasks by leveraging prior experience is a fundamental trait of intelligent systems.

By El Mahdi Chayti, Martin Jaggi

llms efficiency safety

Related stories