arXiv Machine Learning June 2, 2026 By Jianhao Xu, Zhuang Yang

Beyond $\ell_2$-norm and $\ell_\infty$-norm: A Curvature-Inspired $\ell_p$-Norm Scheme for Deep Neural Networks

Read the original on arXiv Machine Learning →

arXiv:2606. 02078v1 Announce Type: new Abstract: The existing optimizers for deep neural networks (DNNs) typically rely on either the $\ell_2$ norm or the $\ell_\infty$ norm, resulting in optimizers that do not adapt well to substantial changes in curvature across parameter dimensions.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

benchmarks

arXiv Machine Learning

Aug 11

Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.

By Binchuan Qi

safety

arXiv Machine Learning

Jun 2

Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise

arXiv:2605. 18528v2 Announce Type: replace-cross Abstract: A growing lesson from neural network optimization is that optimizer design should respect how the model is parametrized.

By Jiayu Zhang, Tianyi Lin

arXiv Machine Learning

1d ago

On the Principles Behind Neural Network Optimizers

arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.

By Yushun Zhang

llms efficiency

arXiv Machine Learning

Jun 9

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.

By Ganzhao Yuan

arXiv Machine Learning

Jul 7

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

arXiv:2607. 04233v1 Announce Type: cross Abstract: Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems.

By Shokhrukh Ibragimov, Arnulf Jentzen

arXiv Machine Learning

Jun 9

On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature

arXiv:2602. 05600v2 Announce Type: replace Abstract: Stochastic Gradient Descent (SGD) introduces anisotropic noise that is correlated with the local curvature of the loss landscape, thereby biasing optimization toward flat minima.

By Yikuan Zhang, Ning Yang, Yuhai Tu

Related stories

Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise

On the Principles Behind Neural Network Optimizers

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature