arXiv Machine Learning
Subspace Levenberg Marquardt Algorithms in Training Neural Networks
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
Sep 1
Subspace Levenberg Marquardt Algorithms in Training Neural Networks
The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-sized neural networks (NNs). However, its computat...
arXiv Machine Learning
Aug 27
Adaptive Hybrid Subspace Levenberg Marquardt Algorithm with Adequacy Monitor for Large Scale Least Squares Problems
The paper introduces an Adaptive Hybrid Subspace Levenberg–Marquardt (HSLM) algorithm that tackles large‑scale nonlinear least‑squares problems by building a low‑dimensional subspace from gradient, memory, Krylov‑subspace, and randomized curvature data. It employs a deterministic adequacy monitor to adaptively enrich the subspace and decouples step acceptance from damping adjustment, using Armijo backtracking for step length and a ratio of actual to predicted reduction for damping updates. The authors prove global convergence to stationarity and local linear and superlinear convergence, and demonstrate that HSLM matches the convergence of classical and Krylov‑subspace LM while significantly reducing per‑iteration cost, especially as the parameter dimension increases.
arXiv Machine Learning
Sep 4
Linearized subspace refinement framework to expose hidden accuracy in trained neural networks
The paper introduces Linearized Subspace Refinement (LSR), a post‑training framework that uses the local linearized model of a trained neural network to compute a low‑dimensional correction via a reduced least‑squares problem. LSR is architecture‑agnostic and improves accuracy across tasks such as function approximation, operator learning, physics‑informed fine‑tuning, and noisy inverse problems, often achieving order‑of‑magnitude error reductions. The method reveals that standard training can leave significant accuracy plateaus due to numerical ill‑conditioning, and it offers a subspace rank that balances correction strength, stability, and noise sensitivity.
arXiv Machine Learning
Sep 30
Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interest
arXiv:2609.37308v1 Announce Type: cross Abstract: This paper introduces a hybrid joint-selective optimization (HJSO) framework for large-scale numerical problems in which a small subset of trainable...
arXiv Machine Learning
Oct 19
Scalable Levenberg-Marquardt Training Framework for Deep Learning
The paper introduces LeMA, a distributed Levenberg-Marquardt algorithm designed for large-scale neural network training on multi‑GPU systems. By deriving an exact dual formulation that eliminates memory dependence on parameter count and implementing a Jacobian sharding strategy, LeMA reduces the memory footprint from quadratic in parameters to quadratic in batch size. This enables exact LMA steps for billion‑parameter models, achieving a 7.97× speedup and 99.7% strong scaling efficiency on an 8‑GPU node for a convolutional network with over a billion parameters.
arXiv Machine Learning
Sep 17
Gradient Descent with Stochastic Subspaces via Persistence of Memory
The paper introduces a novel technique called "persistence of memory" to enhance stochastic subspace methods for large‑scale optimisation. By using a weakly correlated guidance vector that is refreshed only at wide intervals, the method provides a structured direction for random subspace descent. The authors demonstrate that this guidance can be efficiently computed in sparse or minibatch settings and present the first theoretical analysis of classical SSD methods for sparse functions, showing alignment with low‑lying Hessian eigenvectors near the optimum.