arXiv:2606. 30509v1 Announce Type: new Abstract: Matrix factorization (i.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
arXiv:2609.39595v1 Announce Type: new Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...
The paper introduces low‑rank orthogonalization, a technique that exploits the low‑rank nature of gradients in neural network training to perform matrix orthogonalization more efficiently. Building on this, the authors present low‑rank matrix‑signed gradient descent (MSGD) and a low‑rank variant of the Muon optimizer, showing through experiments that low‑rank Muon matches or surpasses vanilla Muon on GPT‑2 and LLaMA pretraining, especially for larger models. Theoretical analysis provides iteration‑complexity bounds for both low‑rank MSGD and low‑rank Muon under heavy‑tailed noise.
arXiv:2609.07597v1 Announce Type: cross Abstract: Muon can be interpreted as optimizing a linear local objective over a spectral-norm ball. This gives a matrix-sign update that preserves the singular...
arXiv:2607. 20512v1 Announce Type: cross Abstract: The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW.
The paper presents the first asymptotic convergence guarantees for the Muon algorithm, showing that with suitable hyperparameters the iterates satisfy ≠≠ ∥∇f(x_k)∥ → 0 and, under a global Polyak-ℒojasiewicz condition, the function values converge linearly. It reveals that Muon’s implicit regularization acts as a bounded preconditioner, framing Muon as a preconditioned Polyak heavy‑ball method and enabling a Lyapunov analysis. Building on this insight, the authors introduce Muesterov, a Nesterov‑based variant, and prove it shares the same convergence guarantees, extending the theory beyond the heavy‑ball setting; numerical experiments on a scalar cross‑entropy problem and preliminary nanoGPT simulations support the theoretical findings.
arXiv:2602. 05725v3 Announce Type: replace Abstract: Muon updates matrix parameters via the matrix sign of the gradient and has shown strong empirical gains, yet its dynamics and scaling behavior remain unclear in theory.