← Back to all news
arXiv Machine Learning By Vishrut Goyal, Rohan Ramkumar

Spectrally Targeted Muon

Read the original on arXiv Machine Learning →

The paper introduces Spectrally Targeted Muon, an optimizer that orthogonalizes only singular values of update matrices above or below a threshold τ, thereby interpolating between normalized SGD and the full Muon optimizer. It uses Newton‑Schulz iteration on a shifted Gram matrix to avoid SVD, and evaluates the method on CIFAR‑10 and NanoGPT, tracking effective rank and a new alignment metric. The study finds that orthogonalizing small singular values is crucial, that shrinking large singular values keeps the parameter spectrum flat, and that AdamW differs mainly in the speed of structure formation compared to Muon.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 25

A Physical Response-and-Memory Model for Muon Optimization

The paper introduces a physical response-and-memory model for the Muon optimizer, explaining its semi‑orthogonalized momentum update as the maximally dissipative direction under an output‑side safety budget. It treats the weight matrix as a responsive medium with internal stress, showing that momentum corresponds to accumulated stress whose relaxation occurs over multiple timescales—fast and slow. Based on this, the authors propose the Bi‑Maxwell optimizer, which uses a two‑timescale memory kernel and achieves target loss in fewer steps on a public large‑language‑model benchmark.

By Yinze Hu, Hongjun Xiang, Xingao Gong, Hongyu Yu
llmsbenchmarkssafety
More like this →
arXiv Machine Learning
Aug 27

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.

By Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland
llms
More like this →
arXiv AI
Jul 16

Reassessing Muon for Matrix Factorization

arXiv:2607. 13246v1 Announce Type: cross Abstract: Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training.

By Ali Parviz, Gal Mishne, Alex Cloninger
llmsbenchmarks
More like this →