arXiv:2607. 06976v1 Announce Type: cross Abstract: In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training. This method generalizes the LSGD method for DeepONets.
The article presents a detailed, step‑by‑step derivation of the backpropagation algorithm for a fully connected feedforward neural network with sigmoid activations and squared‑error loss. It starts with a single‑weight example, extends to general error terms and matrix equations, and then scales the derivation to multiple training examples and mini‑batch gradient descent. Each equation is annotated with dimensions and linked to a NumPy implementation trained on handwritten Arabic digits, with full code provided.
The paper introduces low‑rank orthogonalization, a technique that exploits the low‑rank nature of gradients in neural network training to perform matrix orthogonalization more efficiently. Building on this, the authors present low‑rank matrix‑signed gradient descent (MSGD) and a low‑rank variant of the Muon optimizer, showing through experiments that low‑rank Muon matches or surpasses vanilla Muon on GPT‑2 and LLaMA pretraining, especially for larger models. Theoretical analysis provides iteration‑complexity bounds for both low‑rank MSGD and low‑rank Muon under heavy‑tailed noise.
arXiv:2609.00789v1 Announce Type: new Abstract: The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-s...
The paper introduces LeMA, a distributed Levenberg-Marquardt algorithm designed for large-scale neural network training on multi‑GPU systems. By deriving an exact dual formulation that eliminates memory dependence on parameter count and implementing a Jacobian sharding strategy, LeMA reduces the memory footprint from quadratic in parameters to quadratic in batch size. This enables exact LMA steps for billion‑parameter models, achieving a 7.97× speedup and 99.7% strong scaling efficiency on an 8‑GPU node for a convolutional network with over a billion parameters.
arXiv:2512. 17473v3 Announce Type: replace-cross Abstract: We present an algorithm based on the alternating direction method of multipliers (ADMM) for solving nonlinear matrix decompositions (NMD).