← Back to all news
arXiv Machine Learning

A Quadratic Lens on Muon's Polar Update: Benefits, Pitfalls, and the Role of Newton-Schulz

The paper investigates PolarGD, the orthogonalized update used in the Muon optimizer, by analyzing its behavior on matrix quadratic problems where closed‑form solutions are available. It shows that PolarGD can reach the minimizer in a single step when error rows are orthogonal with equal norms, and that it behaves like gradient descent preconditioned by the current iterate, making its speed depend on the iterate’s conditioning rather than the Hessian. The study also compares PolarGD to sign descent and Newton‑Schulz iterations, finding that fewer Newton‑Schulz steps can sometimes converge faster, especially when many singular values of attention matrices are small.

Related stories

arXiv Machine Learning
Oct 9

Spectrally Targeted Muon

The paper introduces Spectrally Targeted Muon, an optimizer that orthogonalizes only singular values of update matrices above or below a threshold τ, thereby interpolating between normalized SGD and the full Muon optimizer. It uses Newton‑Schulz iteration on a shifted Gram matrix to avoid SVD, and evaluates the method on CIFAR‑10 and NanoGPT, tracking effective rank and a new alignment metric. The study finds that orthogonalizing small singular values is crucial, that shrinking large singular values keeps the parameter spectrum flat, and that AdamW differs mainly in the speed of structure formation compared to Muon.

By Vishrut Goyal, Rohan Ramkumar
llmssafety
More like this →
arXiv Machine Learning
Jun 16

CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor

arXiv:2606. 16371v1 Announce Type: new Abstract: Muon is an optimizer that computes updates using the polar factor of the momentum matrix and has shown strong empirical performance across a range of training settings.

By Bishnu Dev (Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE), Sushil Bohara (Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE), Martin Tak\'a\v{c} (Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE), Samuel Horv\'ath (Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE)
More like this →
arXiv AI
Aug 25

A Physical Response-and-Memory Model for Muon Optimization

The paper introduces a physical response-and-memory model for the Muon optimizer, explaining its semi‑orthogonalized momentum update as the maximally dissipative direction under an output‑side safety budget. It treats the weight matrix as a responsive medium with internal stress, showing that momentum corresponds to accumulated stress whose relaxation occurs over multiple timescales—fast and slow. Based on this, the authors propose the Bi‑Maxwell optimizer, which uses a two‑timescale memory kernel and achieves target loss in fewer steps on a public large‑language‑model benchmark.

By Yinze Hu, Hongjun Xiang, Xingao Gong, Hongyu Yu
llmsbenchmarkssafety
More like this →
arXiv Machine Learning
Sep 18

Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training

The paper introduces low‑rank orthogonalization, a technique that exploits the low‑rank nature of gradients in neural network training to perform matrix orthogonalization more efficiently. Building on this, the authors present low‑rank matrix‑signed gradient descent (MSGD) and a low‑rank variant of the Muon optimizer, showing through experiments that low‑rank Muon matches or surpasses vanilla Muon on GPT‑2 and LLaMA pretraining, especially for larger models. Theoretical analysis provides iteration‑complexity bounds for both low‑rank MSGD and low‑rank Muon under heavy‑tailed noise.

By Chuan He, Zhanwang Deng, Zhaosong Lu
llms
More like this →