The paper introduces a least‑squares method for training quadratic neural networks with regularization, providing a lower bound on the training optimization problem when the regularization coefficient is positive. It delivers closed‑form expressions for both the approximate solution and its sensitivity to data errors, and shows that the solution is optimal when the regularization coefficient is zero. The approach offers computational advantages over iterative methods like backpropagation and is validated on a nonlinear system identification example.
arXiv:2502. 11152v4 Announce Type: replace-cross Abstract: The optimization foundations of deep linear networks have recently received significant attention.
arXiv:2403. 10232v2 Announce Type: replace-cross Abstract: Conventional matrix completion methods approximate the missing values by assuming the matrix to be low-rank, which leads to a linear approximation of missing values.
arXiv:2609.00789v1 Announce Type: new Abstract: The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-s...
The paper presents a near-complete, nonasymptotic generalization theory for multilayer neural networks using path regularization, applicable to broad Lipschitz loss functions without requiring bounded loss or extreme network hyperparameters. It provides an explicit upper bound that addresses approximation rates in generalized Barron spaces and demonstrates the double descent phenomenon for ReLU networks. The authors claim near-minimax optimality for regression problems and plan to establish matching lower bounds in future work.
arXiv:2606. 00340v1 Announce Type: new Abstract: We study optimal learning-rate selection in two-layer and three-layer linear neural networks trained to learn linear target functions.
arXiv:2601. 07397v2 Announce Type: replace-cross Abstract: In this work, we propose a novel layerwise adaptive construction method for neural network architectures.
arXiv:2409. 08066v3 Announce Type: replace Abstract: The real-time solution of parametric optimization problems is critical for applications that demand high accuracy under tight real-time constraints, such as model predictive control.
The paper introduces Linearized Subspace Refinement (LSR), a post‑training framework that uses the local linearized model of a trained neural network to compute a low‑dimensional correction via a reduced least‑squares problem. LSR is architecture‑agnostic and improves accuracy across tasks such as function approximation, operator learning, physics‑informed fine‑tuning, and noisy inverse problems, often achieving order‑of‑magnitude error reductions. The method reveals that standard training can leave significant accuracy plateaus due to numerical ill‑conditioning, and it offers a subspace rank that balances correction strength, stability, and noise sensitivity.
The paper proposes an analytic method for determining the optimal early‑stopping time in training neural networks, avoiding the need for gradient‑descent training. It uses Rademacher complexity with an L1‑norm to estimate generalization error, offering a more general approach than previous random‑matrix‑theory based methods. The framework is demonstrated on linear regression and extended to nonlinear neural networks via linear probing, as shown in a MNIST classification example.
Training neural networks requires balancing the trade-off between fitting the training data and achieving robust performance on unseen inputs. This ability, commonly referred to as generalizability, i...
The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-sized neural networks (NNs). However, its computat...