The paper analyzes the stochastic optimizer Muon, which orthogonalizes an exponential moving average of minibatch gradients, and compares its behavior to the baseline SignSGD on a high‑dimensional matrix‑valued least squares problem. It derives deterministic dynamics showing Muon’s sensitivity to an effective batch size and its failure to resolve small eigenmodes when whitening is exact and the effective batch is small, while Momentum and finite Newton–Shulz iterations can mitigate this. The study presents a phase diagram in terms of data and target exponents, indicating when Muon or SignSGD is preferable based on covariance spectrum steepness and target loss scaling.
By Elliot Paquette, Noah Marshall, Lucas Benigni, Guangyuan Wang, Atish Agarwala, Courtney Paquette
arXiv:2609.07597v1 Announce Type: cross
Abstract: Muon can be interpreted as optimizing a linear local objective over a spectral-norm ball. This gives a matrix-sign update that preserves the singular...
By Qiaozhe Zhang, Jun Sun, Yingzhuang Liu
arXiv:2606. 13867v1 Announce Type: new Abstract: Muon is an increasingly widely used optimizer that replaces a gradient $G=USV^\top$ with its polar factor $UV^\top$, thereby flattening the singular spectrum.
By Yihe Dong, Will Sawin
arXiv:2606. 04058v1 Announce Type: cross Abstract: Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon.
By Gagik Magakyan, Pablo Parrilo, Asuman Ozdaglar
arXiv:2605. 17109v3 Announce Type: replace-cross Abstract: In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly.
By Fangzhou Wu, Rikhav Shah, Sandeep Silwal, Qiuyi Zhang
arXiv:2610.20112v1 Announce Type: new
Abstract: Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (...
By Yongtao Wu, Mahdi Mansouri, Zhenyu Zhu
arXiv:2606. 08388v1 Announce Type: new Abstract: Muon replaces a matrix gradient $G=U\Sigma V^\top$ by its polar factor $UV^\top$.
By Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso Poggio
arXiv:2607. 20512v1 Announce Type: cross Abstract: The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW.
By Yufeng Wang
The paper investigates how much spectral detail is necessary for the Muon optimizer, which traditionally uses a flat spectral profile. By analyzing singular modes, the authors find that most modes lie below a noise edge yet align positively with the gradient, leading them to propose BulkBoost—a two‑band spectral reweighting framework that adjusts bulk and spike gains while preserving matrix norms. Experiments across various pre‑training settings show that this low‑dimensional reweighting matches or surpasses fine‑grained spectral profiles, achieving notable loss reductions.
By Meher Chaitanya, Tianyi Zhou, Aristides Gionis
arXiv:2609.39595v1 Announce Type: new
Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...
By Hanyng Peng, Hui Wang, Yue Yu
The paper introduces Spectrally Targeted Muon, an optimizer that orthogonalizes only singular values of update matrices above or below a threshold τ, thereby interpolating between normalized SGD and the full Muon optimizer. It uses Newton‑Schulz iteration on a shifted Gram matrix to avoid SVD, and evaluates the method on CIFAR‑10 and NanoGPT, tracking effective rank and a new alignment metric. The study finds that orthogonalizing small singular values is crucial, that shrinking large singular values keeps the parameter spectrum flat, and that AdamW differs mainly in the speed of structure formation compared to Muon.
By Vishrut Goyal, Rohan Ramkumar
arXiv:2607. 19771v1 Announce Type: cross Abstract: Muon and related matrix-sign optimizers are increasingly used to pre-train large language models, but their effect on the internal geometry of individual weight matrices is not well understood.
By Jiachun Li