The paper introduces an Adaptive Hybrid Subspace Levenberg–Marquardt (HSLM) algorithm that tackles large‑scale nonlinear least‑squares problems by building a low‑dimensional subspace from gradient, memory, Krylov‑subspace, and randomized curvature data. It employs a deterministic adequacy monitor to adaptively enrich the subspace and decouples step acceptance from damping adjustment, using Armijo backtracking for step length and a ratio of actual to predicted reduction for damping updates. The authors prove global convergence to stationarity and local linear and superlinear convergence, and demonstrate that HSLM matches the convergence of classical and Krylov‑subspace LM while significantly reducing per‑iteration cost, especially as the parameter dimension increases.
arXiv:2609.00789v1 Announce Type: new Abstract: The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-s...
The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-sized neural networks (NNs). However, its computat...
arXiv:2606. 18175v1 Announce Type: cross Abstract: We present a numerical method for the forward solution of nonlinear partial differential equations (PDEs) in which Bellman-Kalaba quasilinearization reduces the nonlinear problem to a sequence of linear subproblems, each discretized by collocation onto a trial space that is linear in its parameters and solved by a single direct linear least-squares QR factorization.
arXiv:2609.37308v1 Announce Type: cross Abstract: This paper introduces a hybrid joint-selective optimization (HJSO) framework for large-scale numerical problems in which a small subset of trainable...
The paper introduces Linearized Subspace Refinement (LSR), a post‑training framework that uses the local linearized model of a trained neural network to compute a low‑dimensional correction via a reduced least‑squares problem. LSR is architecture‑agnostic and improves accuracy across tasks such as function approximation, operator learning, physics‑informed fine‑tuning, and noisy inverse problems, often achieving order‑of‑magnitude error reductions. The method reveals that standard training can leave significant accuracy plateaus due to numerical ill‑conditioning, and it offers a subspace rank that balances correction strength, stability, and noise sensitivity.
arXiv:2610.07211v1 Announce Type: cross Abstract: Systems of equations arise in a wide range of scientific and engineering applications. The present work focuses on solvers for general systems of equ...
Nonlinear least-squares optimization is central to regression, physics-informed neural networks, and other machine-learning tasks. Such problems have a natural geometric interpretation, model predictions form a manifold in data space, while the chosen parameterization can introduce parameter-effects curvature that becomes a dominant source of nonlinearity.
arXiv:2609. 30501v1 Announce Type: new Abstract: Although bilevel optimization (BLO) has emerged as a powerful framework for addressing many complex and nested machine learning problems in recent years, most existing studies are confined to the lower-level strongly convex (LLSC) or lower-level generally convex (LLGC) settings (i.
arXiv:2607. 07623v1 Announce Type: new Abstract: Nonlinear least-squares optimization is central to regression, physics-informed neural networks, and other machine-learning tasks.
arXiv:2608. 05892v1 Announce Type: new Abstract: Physics-informed neural networks (PINNs) often fail to accurately resolve partial differential equations (PDEs) with high-frequency or multi-scale solutions, as well as strongly nonlinear problems.
The paper introduces a novel technique called "persistence of memory" to enhance stochastic subspace methods for large‑scale optimisation. By using a weakly correlated guidance vector that is refreshed only at wide intervals, the method provides a structured direction for random subspace descent. The authors demonstrate that this guidance can be efficiently computed in sparse or minibatch settings and present the first theoretical analysis of classical SSD methods for sparse functions, showing alignment with low‑lying Hessian eigenvectors near the optimum.
arXiv:2605. 04853v2 Announce Type: replace Abstract: We propose HIN-LRI, a hybrid framework that augments a classical numerical solver with a neural operator trained to correct the solver's structured truncation error.