arXiv Machine Learning
Nonsmooth Optimization via Orthogonalized Momentum
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 9
OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality
arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.
arXiv Machine Learning
Oct 15
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
arXiv:2610.19844v1 Announce Type: new Abstract: Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization pe...
arXiv Machine Learning
Oct 15
From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
arXiv:2610.20112v1 Announce Type: new Abstract: Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (...
arXiv AI
Oct 5
Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization
arXiv:2610. 03306v1 Announce Type: cross Abstract: Existing convergence analyses of Muon either assume exact orthogonalization or analyze classical Newton--Schulz polynomials, and guarantee only stationarity, so it is unresolved what Muon's five tuned Newton--Schulz steps preserve and whether that suffices to reach a prescribed neural-network training loss.
arXiv Statistics ML
Oct 28
Phases of Muon: When Muon Eclipses SignSGD
The paper analyzes the stochastic optimizer Muon, which orthogonalizes an exponential moving average of minibatch gradients, and compares its behavior to the baseline SignSGD on a high‑dimensional matrix‑valued least squares problem. It derives deterministic dynamics showing Muon’s sensitivity to an effective batch size and its failure to resolve small eigenmodes when whitening is exact and the effective batch is small, while Momentum and finite Newton–Shulz iterations can mitigate this. The study presents a phase diagram in terms of data and target exponents, indicating when Muon or SignSGD is preferable based on covariance spectrum steepness and target loss scaling.
arXiv Machine Learning
Oct 2
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
arXiv:2609.39595v1 Announce Type: new Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...