arXiv AI
Jul 24
The Active Ingredient in Muon's Grokking
arXiv:2607. 20512v1 Announce Type: cross Abstract: The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW.
Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
arXiv:2606. 30509v1 Announce Type: new Abstract: Matrix factorization (i.
Related stories
arXiv Machine Learning
Jun 4
Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
arXiv:2602. 05725v3 Announce Type: replace Abstract: Muon updates matrix parameters via the matrix sign of the gradient and has shown strong empirical gains, yet its dynamics and scaling behavior remain unclear in theory.
arXiv AI
Jul 23
An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies
arXiv:2607. 19771v1 Announce Type: cross Abstract: Muon and related matrix-sign optimizers are increasingly used to pre-train large language models, but their effect on the internal geometry of individual weight matrices is not well understood.
arXiv Machine Learning
Aug 6
The Loss Does Not See the Basis, but Adam Does
arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.
arXiv Machine Learning
Jun 9
Convergence Bound and Critical Batch Size of Muon Optimizer
arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.
arXiv AI
Jun 12
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
arXiv:2606. 12921v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensitive to initialization choices, its optimal learning rates transfer poorly across ranks, and it often fails to beat dense baselines.
arXiv Machine Learning
Aug 6
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
arXiv:2608. 04607v1 Announce Type: cross Abstract: Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs).
arXiv Machine Learning
Jun 25
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.
arXiv Machine Learning
Jul 27
Hyperball May Not Be a Free Lunch
arXiv:2607. 22444v1 Announce Type: new Abstract: For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates.
arXiv Machine Learning
Jun 9
The Spectral Dynamics and Noise Geometry of Muon
arXiv:2606. 08388v1 Announce Type: new Abstract: Muon replaces a matrix gradient $G=U\Sigma V^\top$ by its polar factor $UV^\top$.
arXiv AI
Jun 9
Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning
arXiv:2604. 09967v2 Announce Type: replace-cross Abstract: Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization.
arXiv Machine Learning
Jun 9
LiMuon: Light and Fast Muon Optimizer for Large Models
arXiv:2509. 14562v4 Announce Type: replace Abstract: Large models recently are widely applied in machine learning, so efficient training of large models has received widespread attention.