arXiv Machine Learning
Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Statistics ML
6d ago
Affine-invariant Cubic Newton with Weak Learners
arXiv:2610.33516v1 Announce Type: new Abstract: Modern gradient boosting algorithms are based on Newton's method, which excels empirically but lacks global convergence guarantees. While recent work o...
arXiv AI
Aug 24
Primal Acceleration of Newton's Method
arXiv:2608. 21359v1 Announce Type: cross Abstract: We develop a new direct accelerated Newton method for minimizing convex functions with Lipschitz continuous Hessian.
arXiv Machine Learning
Jul 28
Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided H\"older Regularity
arXiv:2607. 22906v1 Announce Type: new Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided H\"older regularity.
arXiv Machine Learning
5d ago
Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
The paper introduces LITE, a strategy that accelerates large language model pre‑training by enhancing dynamics along flat directions in the optimization landscape. It builds on a Riemannian ODE framework to interpret adaptive optimizers, then applies larger Hessian damping coefficients and learning rates to improve convergence. Experiments show LITE speeds up Muon and SOAP across various architectures, scales, datasets, and learning‑rate schedules.
arXiv Machine Learning
Oct 12
Second-Order Optimization with Asynchronous Curvature
The paper introduces Split-Client, a task‑parallel scheme that separates Hessian factorization (performed by a curvature client in the background) from gradient updates (performed by a gradient client using the latest available Hessian). It proves that the resulting first‑order stationarity bound depends on the average age of the curvature matrix rather than its maximum, and extends this analysis to subsampled and inexact Hessians. Experiments on two LIBSVM datasets show that this approach reduces the objective faster in wall‑clock time than both Lazy and Vanilla cubic Newton, especially at intermediate accuracy levels.
arXiv Statistics ML
6d ago
A knowledge-driven nonconvex square-root regularization framework for joint group sparsity and structured smoothness
arXiv:2610.35428v1 Announce Type: new Abstract: Structured sparse learning is a leading paradigm for high-dimensional data modeling, as it enables rigorous integration of domain-validated structural...