The paper investigates the last-iterate convergence properties of the Muon optimization algorithm, which employs a fixed number of Newton–Schulz iterations for large-scale language‑model training. It shows that without input stabilization Muon can cycle even on smooth convex quadratics, while with positive stabilization and a momentum coefficient below one, last‑iterate gradient convergence is achieved under a small constant learning rate. For stochastic gradients, the authors demonstrate that fixed learning rates can fail, but decreasing learning rates combined with increasing momentum yield almost‑sure and mean‑square convergence, and they provide counterexamples and density‑one convergence rates.
Muon has achieved strong empirical performance in large-scale language-model training, motivating a better theoretical understanding of its practical implementation. Unlike the idealized polar-factor...
arXiv:2609.39595v1 Announce Type: new Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...
The paper introduces Muon, an optimizer that uses a finite number of Newton‑Schulz iterations to approximate the polar factor for matrix‑valued parameters in large language model pretraining. It demonstrates that this finite iteration smooths the discontinuous polar map into a Lipschitz function of singular values, enabling a conversion from online learning regret to a stationarity guarantee in nonsmooth nonconvex optimization. The authors prove that a logarithmic depth in Newton‑Schulz suffices for convergence to stationary points, matching best‑known sample complexity bounds and extending the result to other spectral maps with similar smoothing properties.
arXiv:2608. 04607v1 Announce Type: cross Abstract: Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs).
arXiv:2610.20112v1 Announce Type: new Abstract: Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (...
The paper presents the first asymptotic convergence guarantees for the Muon algorithm, showing that with suitable hyperparameters the iterates satisfy ≠≠ ∥∇f(x_k)∥ → 0 and, under a global Polyak-ℒojasiewicz condition, the function values converge linearly. It reveals that Muon’s implicit regularization acts as a bounded preconditioner, framing Muon as a preconditioned Polyak heavy‑ball method and enabling a Lyapunov analysis. Building on this insight, the authors introduce Muesterov, a Nesterov‑based variant, and prove it shares the same convergence guarantees, extending the theory beyond the heavy‑ball setting; numerical experiments on a scalar cross‑entropy problem and preliminary nanoGPT simulations support the theoretical findings.
Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.
arXiv:2509. 14562v4 Announce Type: replace Abstract: Large models recently are widely applied in machine learning, so efficient training of large models has received widespread attention.
arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...
arXiv:2610. 03306v1 Announce Type: cross Abstract: Existing convergence analyses of Muon either assume exact orthogonalization or analyze classical Newton--Schulz polynomials, and guarantee only stationarity, so it is unresolved what Muon's five tuned Newton--Schulz steps preserve and whether that suffices to reach a prescribed neural-network training loss.
The paper analyzes the stochastic optimizer Muon, which orthogonalizes an exponential moving average of minibatch gradients, and compares its behavior to the baseline SignSGD on a high‑dimensional matrix‑valued least squares problem. It derives deterministic dynamics showing Muon’s sensitivity to an effective batch size and its failure to resolve small eigenmodes when whitening is exact and the effective batch is small, while Momentum and finite Newton–Shulz iterations can mitigate this. The study presents a phase diagram in terms of data and target exponents, indicating when Muon or SignSGD is preferable based on covariance spectrum steepness and target loss scaling.
arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.