The paper investigates the last-iterate convergence properties of the Muon optimization algorithm, which employs a fixed number of Newton–Schulz iterations for large-scale language‑model training. It shows that without input stabilization Muon can cycle even on smooth convex quadratics, while with positive stabilization and a momentum coefficient below one, last‑iterate gradient convergence is achieved under a small constant learning rate. For stochastic gradients, the authors demonstrate that fixed learning rates can fail, but decreasing learning rates combined with increasing momentum yield almost‑sure and mean‑square convergence, and they provide counterexamples and density‑one convergence rates.
By Ping Li, Vincent Y. F. Tan
Muon has achieved strong empirical performance in large-scale language-model training, motivating a better theoretical understanding of its practical implementation. Unlike the idealized polar-factor...
arXiv:2609.39595v1 Announce Type: new
Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...
By Hanyng Peng, Hui Wang, Yue Yu
arXiv:2610.20112v1 Announce Type: new
Abstract: Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (...
By Yongtao Wu, Mahdi Mansouri, Zhenyu Zhu
arXiv:2608. 04607v1 Announce Type: cross Abstract: Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs).
By Thang Do, Steffen Dereich, Arnulf Jentzen
The paper presents the first asymptotic convergence guarantees for the Muon algorithm, showing that with suitable hyperparameters the iterates satisfy ≠≠ ∥∇f(x_k)∥ → 0 and, under a global Polyak-ℒojasiewicz condition, the function values converge linearly. It reveals that Muon’s implicit regularization acts as a bounded preconditioner, framing Muon as a preconditioned Polyak heavy‑ball method and enabling a Lyapunov analysis. Building on this insight, the authors introduce Muesterov, a Nesterov‑based variant, and prove it shares the same convergence guarantees, extending the theory beyond the heavy‑ball setting; numerical experiments on a scalar cross‑entropy problem and preliminary nanoGPT simulations support the theoretical findings.
By Arthur C. B. de Oliveira, Dhruv D. Jatkar, Guilherme S. Vicinansa, Eduardo D. Sontag
arXiv:2609.13677v1 Announce Type: cross
Abstract: Modern real application problems involve matrix-valued parameters, yet conventional optimizers treat them as vectors, thereby motivating matrix-aware...
By Lexiao Lai, Tianyi Lin, Jiayu Zhang
The paper investigates PolarGD, the orthogonalized update used in the Muon optimizer, by analyzing its behavior on matrix quadratic problems where closed‑form solutions are available. It shows that PolarGD can reach the minimizer in a single step when error rows are orthogonal with equal norms, and that it behaves like gradient descent preconditioned by the current iterate, making its speed depend on the iterate’s conditioning rather than the Hessian. The study also compares PolarGD to sign descent and Newton‑Schulz iterations, finding that fewer Newton‑Schulz steps can sometimes converge faster, especially when many singular values of attention matrices are small.
By Egor Shulgin, Sam Laing, Antonio Orvieto, Peter Richt\'arik
arXiv:2610. 03306v1 Announce Type: cross Abstract: Existing convergence analyses of Muon either assume exact orthogonalization or analyze classical Newton--Schulz polynomials, and guarantee only stationarity, so it is unresolved what Muon's five tuned Newton--Schulz steps preserve and whether that suffices to reach a prescribed neural-network training loss.
By Amartya Roy, Souvik Chakraborty
Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.
By Zhuanghua Liu, Menglian Wang, Luo Luo
arXiv:2509. 14562v4 Announce Type: replace Abstract: Large models recently are widely applied in machine learning, so efficient training of large models has received widespread attention.
By Feihu Huang, Yuning Luo, Songcan Chen
arXiv:2610. 12328v1 Announce Type: new Abstract: We consider stochastic nonsmooth nonconvex composite optimization, which includes several important problems such as constrained optimization and the regularized training of neural networks.
By Mingyi Li, Taira Tsuchiya, Kenji Yamanishi