arXiv:2610.33516v1 Announce Type: new Abstract: Modern gradient boosting algorithms are based on Newton's method, which excels empirically but lacks global convergence guarantees. While recent work o...
arXiv:2608. 21359v1 Announce Type: cross Abstract: We develop a new direct accelerated Newton method for minimizing convex functions with Lipschitz continuous Hessian.
arXiv:2607. 22906v1 Announce Type: new Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided H\"older regularity.
The paper introduces LITE, a strategy that accelerates large language model pre‑training by enhancing dynamics along flat directions in the optimization landscape. It builds on a Riemannian ODE framework to interpret adaptive optimizers, then applies larger Hessian damping coefficients and learning rates to improve convergence. Experiments show LITE speeds up Muon and SOAP across various architectures, scales, datasets, and learning‑rate schedules.
The paper introduces Split-Client, a task‑parallel scheme that separates Hessian factorization (performed by a curvature client in the background) from gradient updates (performed by a gradient client using the latest available Hessian). It proves that the resulting first‑order stationarity bound depends on the average age of the curvature matrix rather than its maximum, and extends this analysis to subsampled and inexact Hessians. Experiments on two LIBSVM datasets show that this approach reduces the objective faster in wall‑clock time than both Lazy and Vanilla cubic Newton, especially at intermediate accuracy levels.
arXiv:2610.35428v1 Announce Type: new Abstract: Structured sparse learning is a leading paradigm for high-dimensional data modeling, as it enables rigorous integration of domain-validated structural...
arXiv:2603. 10485v3 Announce Type: replace-cross Abstract: In this work, we study the convergence properties of the Dual Space Preconditioned Gradient Descent, encompassing optimizers such as Normalized Gradient Descent and Gradient Clipping.
arXiv:2606. 02078v1 Announce Type: new Abstract: The existing optimizers for deep neural networks (DNNs) typically rely on either the $\ell_2$ norm or the $\ell_\infty$ norm, resulting in optimizers that do not adapt well to substantial changes in curvature across parameter dimensions.
Structured sparse learning is a leading paradigm for high-dimensional data modeling, as it enables rigorous integration of domain-validated structural priors into the learning process. Among these pri...
The paper introduces an Adaptive Hybrid Subspace Levenberg–Marquardt (HSLM) algorithm that tackles large‑scale nonlinear least‑squares problems by building a low‑dimensional subspace from gradient, memory, Krylov‑subspace, and randomized curvature data. It employs a deterministic adequacy monitor to adaptively enrich the subspace and decouples step acceptance from damping adjustment, using Armijo backtracking for step length and a ratio of actual to predicted reduction for damping updates. The authors prove global convergence to stationarity and local linear and superlinear convergence, and demonstrate that HSLM matches the convergence of classical and Krylov‑subspace LM while significantly reducing per‑iteration cost, especially as the parameter dimension increases.
The paper introduces ZFO, a lightweight framework that separates direction selection from step-size determination in large‑scale neural network optimization. ZFO uses a trusted first‑order optimizer to pick a search direction and then performs only two additional objective evaluations to build a local curvature‑aware model, selecting an adaptive step within a bounded interval. The authors provide theoretical guarantees for reliable curvature estimation, near‑optimal step selection, and convergence to a stationary point, and demonstrate that ZFO improves optimization and final performance over fixed‑step first‑order baselines on language‑model fine‑tuning tasks.
The paper introduces Curvature-Scaled Sharpness-Aware Minimization (CS‑SAM), a variant of SAM that scales both the perturbation and descent step by curvature bounds to uniformly control gradient distortion under generalized smoothness conditions. It proves deterministic ≥1/T convergence rates for nonconvex objectives, last‑iterate ≥1/T rates for convex objectives, and linear convergence under the Polyak‑Łojasiewicz condition, while recovering standard gradient descent rates when the perturbation is zero. Experiments on computer vision tasks show that CS‑SAM, particularly the (H0,H1) variant, achieves the highest mean best‑test accuracy across four settings compared to tuned SAM and USAM.