← Back to all news
arXiv Machine Learning By Xianghao Wang, Johan Barth\'elemy, John Taylor

Scalable Levenberg-Marquardt Training Framework for Deep Learning

Read the original on arXiv Machine Learning →

The paper introduces LeMA, a distributed Levenberg-Marquardt algorithm designed for large-scale neural network training on multi‑GPU systems. By deriving an exact dual formulation that eliminates memory dependence on parameter count and implementing a Jacobian sharding strategy, LeMA reduces the memory footprint from quadratic in parameters to quadratic in batch size. This enables exact LMA steps for billion‑parameter models, achieving a 7.97× speedup and 99.7% strong scaling efficiency on an 8‑GPU node for a convolutional network with over a billion parameters.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 12

BLOKS: Scaling Kernel Solvers with Sketched Block Preconditioning

BLOKS is a Conjugate Gradient-based solver for Nyström‑sketched Kernel Ridge Regression that introduces a novel preconditioner combining block Jacobi approximation with sketching. This preconditioner’s memory cost is independent of dataset size and model parameters, enabling scaling to large models while maintaining GPU acceleration. Experiments on standard large‑scale datasets show BLOKS achieves accuracy comparable to or better than state‑of‑the‑art kernel solvers with favorable time/memory complexity.

By Giordano Vitale, Giacomo Meanti, Marco Rando, Blanca Ayuso de Dios, Lorenzo Rosasco
benchmarks
More like this →
arXiv AI
Jun 15

Gefen: Optimized Stochastic Optimizer

arXiv:2606. 13894v1 Announce Type: cross Abstract: AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory.

By Nadav Benedek, Tomer Koren, Ohad Fried
efficiency
More like this →
arXiv Machine Learning
Oct 16

The importance of tensor handling and implementation of physics-informed DeepONets

The paper presents a modified physics-informed DeepONet (PI-DeepONet) architecture that reduces computational cost by eliminating redundancy in the Branch network’s handling of input functions across collocation points. The new formulation is mathematically equivalent to the original but uses smaller input tensors and fewer network evaluations, leading to faster training while maintaining comparable predictive accuracy on 1D, 2D, and 3D linear advection problems. Experiments show up to a 3.75× speed‑up in 2D and enable feasible training of 3D models on an NVIDIA A100 GPU, where the original architecture fails due to memory limits.

By Nahum Banks, Gustavo A. M. de Almeida, Sergio Maldonado
More like this →