arXiv Machine Learning
Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Statistics ML
3d ago
Phases of Muon: When Muon Eclipses SignSGD
The paper analyzes the stochastic optimizer Muon, which orthogonalizes an exponential moving average of minibatch gradients, and compares its behavior to the baseline SignSGD on a high‑dimensional matrix‑valued least squares problem. It derives deterministic dynamics showing Muon’s sensitivity to an effective batch size and its failure to resolve small eigenmodes when whitening is exact and the effective batch is small, while Momentum and finite Newton–Shulz iterations can mitigate this. The study presents a phase diagram in terms of data and target exponents, indicating when Muon or SignSGD is preferable based on covariance spectrum steepness and target loss scaling.
arXiv AI
Sep 10
Beyond the Matrix Sign: Quadratic Spectral Descent
arXiv:2609.07597v1 Announce Type: cross Abstract: Muon can be interpreted as optimizing a linear local objective over a spectral-norm ball. This gives a matrix-sign update that preserves the singular...
arXiv Machine Learning
Jun 15
Muon$^p$: Muon with Fractional Spectral Powers
arXiv:2606. 13867v1 Announce Type: new Abstract: Muon is an increasingly widely used optimizer that replaces a gradient $G=USV^\top$ with its polar factor $UV^\top$, thereby flattening the singular spectrum.
arXiv AI
Jun 4
Spectral Scaling Laws of Muon
arXiv:2606. 04058v1 Announce Type: cross Abstract: Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon.
arXiv AI
Jun 2
DynMuon: A Dynamic Spectral Shaping View of Muon
arXiv:2605. 17109v3 Announce Type: replace-cross Abstract: In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly.
arXiv Machine Learning
Oct 15
From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
arXiv:2610.20112v1 Announce Type: new Abstract: Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (...