arXiv Machine Learning June 10, 2026 By Ryo Sagawa, Daisuke Furihata, Yuto Miyatake

Accelerating SAV-based optimization via randomized low-rank Hessian approximation

Read the original on arXiv Machine Learning →

arXiv:2606. 10562v1 Announce Type: cross Abstract: We propose a new optimization method, the Nystr\"om-enhanced relaxed scalar auxiliary variable method (N-RSAV), which incorporates curvature information into the RSAV framework to accelerate convergence while preserving an unconditional modified energy dissipation law.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 27

Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers

arXiv:2607. 22004v1 Announce Type: new Abstract: Energy natural gradient descent (ENGD) aligns parameter updates with the curvature of an underlying function-space energy, but existing formulations assume an unconstrained Euclidean parameter domain.

By Zhangyong Liang, Huanhuan Gao

benchmarks

arXiv Machine Learning

Aug 4

Simplified Quadratic Gradient: A Unified Framework Bridging Gradient Descent and Newton-Type Methods by Synthesizing Hessians and Gradients

arXiv:2209. 03282v5 Announce Type: replace-cross Abstract: Accelerating the convergence of second-order optimization, particularly Newton-type methods, remains a pivotal challenge in algorithmic research.

By John Chiang

arXiv Machine Learning

Jul 28

Variational Boosting for Physics-Informed Neural Networks

arXiv:2607. 23940v1 Announce Type: new Abstract: Physics-Informed Neural Networks (PINNs) solve differential equations by minimizing the residual of a nonlinear operator over a neural parameterization of the solution.

By Pavlos Protopapas, Kaylee Vo

safety

arXiv Machine Learning

Jun 16

Schattor: Schatten-family methods for deep learning optimization

arXiv:2606. 15702v1 Announce Type: cross Abstract: Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm design and theoretical analysis.

By Bohao Ma, Junyu Zhang, Chuan He

arXiv Machine Learning

Jul 7

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).

By Steffen Dereich, Arnulf Jentzen, Adrian Riekert

arXiv AI

Jun 3

Introduction to optimization methods for training SciML models

arXiv:2601. 10222v2 Announce Type: replace-cross Abstract: Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains.

By Alena Kopani\v{c}\'akov\'a, Elisa Riccietti

Related stories

Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers

Simplified Quadratic Gradient: A Unified Framework Bridging Gradient Descent and Newton-Type Methods by Synthesizing Hessians and Gradients

Variational Boosting for Physics-Informed Neural Networks

Schattor: Schatten-family methods for deep learning optimization

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

Introduction to optimization methods for training SciML models