A Quadratic Lens on Muon's Polar Update: Benefits, Pitfalls, and the Role of Newton-Schulz

The paper investigates PolarGD, the orthogonalized update used in the Muon optimizer, by analyzing its behavior on matrix quadratic problems where closed‑form solutions are available. It shows that PolarGD can reach the minimizer in a single step when error rows are orthogonal with equal norms, and that it behaves like gradient descent preconditioned by the current iterate, making its speed depend on the iterate’s conditioning rather than the Hessian. The study also compares PolarGD to sign descent and Newton‑Schulz iterations, finding that fewer Newton‑Schulz steps can sometimes converge faster, especially when many singular values of attention matrices are small.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• llms