arXiv Machine Learning June 9, 2026 By Naoki Sato, Hiroki Naganuma, Hideaki Iiduka

Convergence Bound and Critical Batch Size of Muon Optimizer

Read the original on arXiv Machine Learning →

arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

computer-vision

arXiv AI

Jul 28

Scale Weight Decay and Train Better

arXiv:2607. 23777v1 Announce Type: cross Abstract: The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data.

By Anuj Apte

safety

arXiv Machine Learning

Jun 30

To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

arXiv:2603. 00742v2 Announce Type: replace Abstract: While Adam has long been the ubiquitous default optimizer for deep neural networks, Muon has recently seen rapid adoption due to its superior training speed.

By Sara Dragutinovi\'c, Yedi Zhang, Rajesh Ranganath

safety

arXiv AI

Jul 24

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

arXiv:2607. 20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.

By Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, Tijmen Blankevoort

llms

arXiv Machine Learning

Aug 6

On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations

arXiv:2608. 04607v1 Announce Type: cross Abstract: Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs).

By Thang Do, Steffen Dereich, Arnulf Jentzen

llms

arXiv Machine Learning

1d ago

On the Principles Behind Neural Network Optimizers

arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.

By Yushun Zhang

llms efficiency

arXiv Machine Learning

Jun 4

Muon in Associative Memory Learning: Training Dynamics and Scaling Laws

arXiv:2602. 05725v3 Announce Type: replace Abstract: Muon updates matrix parameters via the matrix sign of the gradient and has shown strong empirical gains, yet its dynamics and scaling behavior remain unclear in theory.

By Binghui Li, Kaifei Wang, Han Zhong, Pinyan Lu, Liwei Wang

llms safety

Related stories

Scale Weight Decay and Train Better

To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations

On the Principles Behind Neural Network Optimizers

Muon in Associative Memory Learning: Training Dynamics and Scaling Laws