The limited-memory BFGS (L-BFGS) algorithm is a cornerstone of large-scale optimization due to its linear memory and computational costs. However, in ill-conditioned or non-convex landscapes, the implicit inverse Hessian approximation can suffer from an exploding condition number, leading to numerical instability and degraded convergence.
arXiv:2209. 03282v5 Announce Type: replace-cross Abstract: Accelerating the convergence of second-order optimization, particularly Newton-type methods, remains a pivotal challenge in algorithmic research.
By John Chiang
arXiv:2512. 02494v2 Announce Type: replace Abstract: Differentiable optimization layers enable learning systems to make decisions by solving embedded optimization problems.
By Zihao Zhao, Kai-Chia Mo, Shing-Hei Ho, Brandon Amos, Kai Wang
arXiv:2607. 20769v1 Announce Type: new Abstract: Learning-enabled decision systems often use offline data or computation to reduce online compute cost.
By Shijie Pan, Agustin Castellano, Zeyu Shen, Enrique Mallada
arXiv:2508. 00775v2 Announce Type: replace-cross Abstract: The design of many classical optimization algorithms is driven by the certification of linear convergence rates over classes of optimization problems.
By Andrea Martin, Ian R. Manchester, Luca Furieri
arXiv:2511. 19716v3 Announce Type: replace-cross Abstract: Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise.
By Mitchell Scott, Tianshi Xu, Ziyuan Tang, Alexandra Pichette-Emmons, Qiang Ye, Yousef Saad, Yuanzhe Xi
arXiv:2608. 01552v1 Announce Type: cross Abstract: Quadratic Gradient (QG) is a Newton-type optimization framework that bridges first-order gradient descent and second-order optimization by incorporating curvature information into gradient updates.
By John Chiang
arXiv:2607. 22906v1 Announce Type: new Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided H\"older regularity.
By Arzu Ahmadova, Ismail Huseynov
arXiv:2406. 13041v3 Announce Type: replace Abstract: Lower-bound analyses for nonconvex strongly-concave minimax optimization problems have shown that stochastic first-order algorithms require at least $\mathcal{O}(\varepsilon^{-4})$ sample complexity to find an $\varepsilon$-stationary point.
By Haoyuan Cai, Sulaiman A. Alghunaim, Ali H. Sayed
The paper investigates the "edge of stability" phenomenon in deep learning, where Hessian eigenvalues remain stable above a classically predicted unstable threshold. It shows that many first‑order optimizers, including gradient descent, can violate this stability bound by up to a factor of 21.1, and that this deviation depends systematically on the optimizer used. The authors propose a new stability threshold based on the directional Hessian and gradient‑alignment score, which removes optimizer‑dependent offsets and offers consistent predictions while providing diagnostic tools to understand how optimizers balance temporal and spatial budgets.
By Jaerin Lee, Kyoung Mu Lee
arXiv:2606. 30455v1 Announce Type: new Abstract: The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, ignoring the fact that noise in high-curvature directions has less impact because learning rates are already constrained there.
By Muhammad Hamza (Indian Institute of Technology Kharagpur), Ayush Goel (Indian Institute of Technology Kharagpur)
arXiv:2608. 12009v1 Announce Type: cross Abstract: Bregman proximal stochastic gradient (BPSG) methods bring variance-reduced composite optimization to objectives whose geometry is poorly captured by Euclidean smoothness.
By Chenhan Jin, Shengze Xu, Binghui Xie, Kaiwen Zhou, Fan Jia, James Cheng, Tieyong Zeng