arXiv Machine Learning

A family of spectral conjugate gradient algorithms derived by least-squares approximations based on a modified quasi--Newton update with application to a revised robust binary classification model

arXiv Machine Learning

Jun 9

Neural Legendre-Fenchel transform with Hessian Preconditioning

arXiv:2606. 09077v1 Announce Type: new Abstract: The Legendre-Fenchel (LF) transform is a fundamental tool in convex analysis and machine learning that maps lower semi-continuous functions to their convex conjugates.

By Basile Plus-Gourdon, Frank Nielsen

benchmarks

Hugging Face Trending Papers

Jul 7

On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

The limited-memory BFGS (L-BFGS) algorithm is a cornerstone of large-scale optimization due to its linear memory and computational costs. However, in ill-conditioned or non-convex landscapes, the implicit inverse Hessian approximation can suffer from an exploding condition number, leading to numerical instability and degraded convergence.

benchmarks

arXiv Machine Learning

Aug 4

Simplified Quadratic Gradient: A Unified Framework Bridging Gradient Descent and Newton-Type Methods by Synthesizing Hessians and Gradients

arXiv:2209. 03282v5 Announce Type: replace-cross Abstract: Accelerating the convergence of second-order optimization, particularly Newton-type methods, remains a pivotal challenge in algorithmic research.

By John Chiang

arXiv Machine Learning

Jul 28

Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided H\"older Regularity

arXiv:2607. 22906v1 Announce Type: new Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided H\"older regularity.

By Arzu Ahmadova, Ismail Huseynov

benchmarks

arXiv Machine Learning

Jul 8

On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

arXiv:2607. 05836v1 Announce Type: cross Abstract: The limited-memory BFGS (L-BFGS) algorithm is a cornerstone of large-scale optimization due to its linear memory and computational costs.

By Don Li

benchmarks

arXiv Machine Learning

Aug 4

Generalized Quadratic Gradient: A New Direction in Optimization via the Fusion of Positive-Definite Curvature Matrices and Gradients into A Unified Framework

arXiv:2608. 01552v1 Announce Type: cross Abstract: Quadratic Gradient (QG) is a Newton-type optimization framework that bridges first-order gradient descent and second-order optimization by incorporating curvature information into gradient updates.

By John Chiang

arXiv Machine Learning

Jun 30

Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules

arXiv:2606. 30455v1 Announce Type: new Abstract: The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, ignoring the fact that noise in high-curvature directions has less impact because learning rates are already constrained there.

By Muhammad Hamza (Indian Institute of Technology Kharagpur), Ayush Goel (Indian Institute of Technology Kharagpur)

arXiv AI

Sep 10

Beyond the Matrix Sign: Quadratic Spectral Descent

arXiv:2609.07597v1 Announce Type: cross Abstract: Muon can be interpreted as optimizing a linear local objective over a spectral-norm ball. This gives a matrix-sign update that preserves the singular...

By Qiaozhe Zhang, Jun Sun, Yingzhuang Liu

llms

arXiv Machine Learning

Aug 6

On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations

arXiv:2608. 04607v1 Announce Type: cross Abstract: Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs).

By Thang Do, Steffen Dereich, Arnulf Jentzen

llms

arXiv Machine Learning

Aug 5

Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability

arXiv:2511. 19716v3 Announce Type: replace-cross Abstract: Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise.

By Mitchell Scott, Tianshi Xu, Ziyuan Tang, Alexandra Pichette-Emmons, Qiang Ye, Yousef Saad, Yuanzhe Xi

benchmarks

arXiv Machine Learning

Sep 17

Gradient Descent with Stochastic Subspaces via Persistence of Memory

The paper introduces a novel technique called "persistence of memory" to enhance stochastic subspace methods for large‑scale optimisation. By using a weakly correlated guidance vector that is refreshed only at wide intervals, the method provides a structured direction for random subspace descent. The authors demonstrate that this guidance can be efficiently computed in sparse or minibatch settings and present the first theoretical analysis of classical SSD methods for sparse functions, showing alignment with low‑lying Hessian eigenvectors near the optimum.

By Subhroshekhar Ghosh, Clement Z. Q. Ng, Pierre-Louis Poirion, Akiko Takeda

efficiency safety

arXiv Machine Learning

Sep 10

GPU-Enabled Large-Scale Optimization Using Randomized Linear Algebra

arXiv:2609. 08136v1 Announce Type: new Abstract: This paper introduces rlaopt, a PyTorch-based package for large-scale optimization and scientific computing using randomized numerical linear algebra (RandNLA).

By Pratik Rathore, Zachary Frangella, Parth Nobel, Xuning Hu, Madeleine Udell

Related stories