arXiv:2607. 06976v1 Announce Type: cross Abstract: In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training.
In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training. This method generalizes the LSGD method for DeepONets.
The article presents a detailed, step‑by‑step derivation of the backpropagation algorithm for a fully connected feedforward neural network with sigmoid activations and squared‑error loss. It starts with a single‑weight example, extends to general error terms and matrix equations, and then scales the derivation to multiple training examples and mini‑batch gradient descent. Each equation is annotated with dimensions and linked to a NumPy implementation trained on handwritten Arabic digits, with full code provided.
The paper introduces low‑rank orthogonalization, a technique that exploits the low‑rank nature of gradients in neural network training to perform matrix orthogonalization more efficiently. Building on this, the authors present low‑rank matrix‑signed gradient descent (MSGD) and a low‑rank variant of the Muon optimizer, showing through experiments that low‑rank Muon matches or surpasses vanilla Muon on GPT‑2 and LLaMA pretraining, especially for larger models. Theoretical analysis provides iteration‑complexity bounds for both low‑rank MSGD and low‑rank Muon under heavy‑tailed noise.
arXiv:2609.00789v1 Announce Type: new Abstract: The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-s...
The paper introduces LeMA, a distributed Levenberg-Marquardt algorithm designed for large-scale neural network training on multi‑GPU systems. By deriving an exact dual formulation that eliminates memory dependence on parameter count and implementing a Jacobian sharding strategy, LeMA reduces the memory footprint from quadratic in parameters to quadratic in batch size. This enables exact LMA steps for billion‑parameter models, achieving a 7.97× speedup and 99.7% strong scaling efficiency on an 8‑GPU node for a convolutional network with over a billion parameters.
arXiv:2512. 17473v3 Announce Type: replace-cross Abstract: We present an algorithm based on the alternating direction method of multipliers (ADMM) for solving nonlinear matrix decompositions (NMD).
arXiv:2407. 02362v3 Announce Type: replace-cross Abstract: Modern deep neural networks heavily rely on a large number of multiply-accumulate operations, which constitute the predominant computational cost.
The paper introduces a least‑squares method for training quadratic neural networks with regularization, providing a lower bound on the training optimization problem when the regularization coefficient is positive. It delivers closed‑form expressions for both the approximate solution and its sensitivity to data errors, and shows that the solution is optimal when the regularization coefficient is zero. The approach offers computational advantages over iterative methods like backpropagation and is validated on a nonlinear system identification example.
arXiv:2605. 09075v2 Announce Type: replace-cross Abstract: Although the Laplace approximation offers a simple route to uncertainty quantification in deep neural networks, its reliance on inverting large Hessian matrices has motivated a range of computationally feasible low-dimensional or sparse approximations.
arXiv:2606. 02328v1 Announce Type: new Abstract: We explore Riemannian optimization techniques for rank-factored matrix parameters, targeting contemporary deep learning applications.
arXiv:2607. 20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.
arXiv:2607. 16612v1 Announce Type: cross Abstract: Backpropagation makes training deep networks memory intensive because it must store intermediate activations.