arXiv Machine Learning
From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 2
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
arXiv:2609.39595v1 Announce Type: new Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...
arXiv Statistics ML
3d ago
Phases of Muon: When Muon Eclipses SignSGD
The paper analyzes the stochastic optimizer Muon, which orthogonalizes an exponential moving average of minibatch gradients, and compares its behavior to the baseline SignSGD on a high‑dimensional matrix‑valued least squares problem. It derives deterministic dynamics showing Muon’s sensitivity to an effective batch size and its failure to resolve small eigenmodes when whitening is exact and the effective batch is small, while Momentum and finite Newton–Shulz iterations can mitigate this. The study presents a phase diagram in terms of data and target exponents, indicating when Muon or SignSGD is preferable based on covariance spectrum steepness and target loss scaling.
arXiv AI
Oct 5
Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization
arXiv:2610. 03306v1 Announce Type: cross Abstract: Existing convergence analyses of Muon either assume exact orthogonalization or analyze classical Newton--Schulz polynomials, and guarantee only stationarity, so it is unresolved what Muon's five tuned Newton--Schulz steps preserve and whether that suffices to reach a prescribed neural-network training loss.
arXiv Machine Learning
Sep 16
Nonsmooth Optimization via Orthogonalized Momentum
arXiv:2609.13677v1 Announce Type: cross Abstract: Modern real application problems involve matrix-valued parameters, yet conventional optimizers treat them as vectors, thereby motivating matrix-aware...
arXiv Machine Learning
Sep 11
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.
arXiv Machine Learning
3d ago
Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations
The paper investigates the last-iterate convergence properties of the Muon optimization algorithm, which employs a fixed number of Newton–Schulz iterations for large-scale language‑model training. It shows that without input stabilization Muon can cycle even on smooth convex quadratics, while with positive stabilization and a momentum coefficient below one, last‑iterate gradient convergence is achieved under a small constant learning rate. For stochastic gradients, the authors demonstrate that fixed learning rates can fail, but decreasing learning rates combined with increasing momentum yield almost‑sure and mean‑square convergence, and they provide counterexamples and density‑one convergence rates.