← Back to all news
arXiv Machine Learning

Second-Order Optimization with Asynchronous Curvature

The paper introduces Split-Client, a task‑parallel scheme that separates Hessian factorization (performed by a curvature client in the background) from gradient updates (performed by a gradient client using the latest available Hessian). It proves that the resulting first‑order stationarity bound depends on the average age of the curvature matrix rather than its maximum, and extends this analysis to subsampled and inexact Hessians. Experiments on two LIBSVM datasets show that this approach reduces the objective faster in wall‑clock time than both Lazy and Vanilla cubic Newton, especially at intermediate accuracy levels.

Related stories

arXiv Machine Learning
Aug 12

A lower bound for stepsize-based acceleration of gradient descent

arXiv:2608. 10418v1 Announce Type: cross Abstract: Recent work has shown that, for smooth convex optimization, plain gradient descent can be accelerated from its textbook convergence rate of $O(T^{-1})$ (where $T$ denotes the number of iterations) to $O\big(T^{-\log_2(1+\sqrt{2})}\big)$ using carefully designed stepsize schedules alone, without resorting to momentum or other algorithmic modifications.

By Jianhao Ma, Yuxin Chen
llmsbenchmarks
More like this →
Hugging Face Trending Papers
Jul 7

On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

The limited-memory BFGS (L-BFGS) algorithm is a cornerstone of large-scale optimization due to its linear memory and computational costs. However, in ill-conditioned or non-convex landscapes, the implicit inverse Hessian approximation can suffer from an exploding condition number, leading to numerical instability and degraded convergence.

benchmarks
More like this →
arXiv Machine Learning
Oct 3

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

The paper introduces ZFO, a lightweight framework that separates direction selection from step-size determination in large‑scale neural network optimization. ZFO uses a trusted first‑order optimizer to pick a search direction and then performs only two additional objective evaluations to build a local curvature‑aware model, selecting an adaptive step within a bounded interval. The authors provide theoretical guarantees for reliable curvature estimation, near‑optimal step selection, and convergence to a stationary point, and demonstrate that ZFO improves optimization and final performance over fixed‑step first‑order baselines on language‑model fine‑tuning tasks.

By Cristian McGee, El Houcine Bergou, Aritra Dutta
llmsfine-tuning
More like this →
arXiv Statistics ML
6d ago

Affine-invariant Cubic Newton with Weak Learners

arXiv:2610.33516v1 Announce Type: new Abstract: Modern gradient boosting algorithms are based on Newton's method, which excels empirically but lacks global convergence guarantees. While recent work o...

By Nikita Zozoulenko, Daniel Falkowski, Thomas Cass, Lukas Gonon
More like this →
arXiv Machine Learning
Oct 5

Near-Optimal Convex Optimization with Lazy Second-Order Oracles

arXiv:2610. 03222v1 Announce Type: cross Abstract: This paper studies the complexity of convex optimization using lazy second-order oracles (Doikov, Chayti, and Jaggi, ICML 2023), where an algorithm queries gradients every iteration and Hessians once per $m$ iterations.

By Xinliang Zhang, Lesi Chen, Chengchang Liu, Jingzhao Zhang
More like this →