arXiv Machine Learning June 2, 2026 By Jianhao Xu, Zhuang Yang

Beyond $\ell_2$-norm and $\ell_\infty$-norm: A Curvature-Inspired $\ell_p$-Norm Scheme for Deep Neural Networks

Read the original on arXiv Machine Learning →

arXiv:2606. 02078v1 Announce Type: new Abstract: The existing optimizers for deep neural networks (DNNs) typically rely on either the $\ell_2$ norm or the $\ell_\infty$ norm, resulting in optimizers that do not adapt well to substantial changes in curvature across parameter dimensions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

benchmarks

arXiv Machine Learning

Aug 11

Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.

By Binchuan Qi

safety

arXiv Machine Learning

Jun 2

Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise

arXiv:2605. 18528v2 Announce Type: replace-cross Abstract: A growing lesson from neural network optimization is that optimizer design should respect how the model is parametrized.

By Jiayu Zhang, Tianyi Lin

arXiv Statistics ML

1d ago

Exact information accounting for SGD methods

arXiv:2610.00446v1 Announce Type: cross Abstract: As an alternative to the standard geometric analyses, we give an exact, information-theoretic analysis of stochastic gradient descent (SGD) and its v...

By Akshay Balsubramani

arXiv Machine Learning

Aug 18

On the Principles Behind Neural Network Optimizers

arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.

By Yushun Zhang

llms efficiency

arXiv Machine Learning

Jun 9

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.

By Ganzhao Yuan

arXiv Machine Learning

Jul 7

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

arXiv:2607. 04233v1 Announce Type: cross Abstract: Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems.

By Shokhrukh Ibragimov, Arnulf Jentzen

One email a morning, machine-written

Related stories

Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise

Exact information accounting for SGD methods

On the Principles Behind Neural Network Optimizers

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks