← Back to all news
arXiv Machine Learning By Zhuanghua Liu, Menglian Wang, Luo Luo

Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

Read the original on arXiv Machine Learning →

Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jun 4

Spectral Scaling Laws of Muon

arXiv:2606. 04058v1 Announce Type: cross Abstract: Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon.

By Gagik Magakyan, Pablo Parrilo, Asuman Ozdaglar
llmsbenchmarks
More like this →
arXiv Machine Learning
Aug 27

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.

By Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland
llms
More like this →
arXiv Machine Learning
Jun 9

Convergence Bound and Critical Batch Size of Muon Optimizer

arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.

By Naoki Sato, Hiroki Naganuma, Hideaki Iiduka
computer-vision
More like this →
arXiv Machine Learning
Jun 29

Aurora: A Leverage-Aware Spectral Optimizer

arXiv:2606. 27715v1 Announce Type: new Abstract: We show that for tall matrix parameters, like projection matrices in the MLP layers, the Muon update can have row norms that are arbitrarily non-uniform.

By Alec Dewulf, Dhruv Pai, Li Yang, Ashley Zhang, Ben Keigwin
benchmarks
More like this →
arXiv Machine Learning
Jul 17

Muse: Representation Geometry of Muon Beyond Normalized Momentum

arXiv:2607. 14536v1 Announce Type: new Abstract: Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization.

By Da Chang, Qiankun Shi, Lvgang Zhang, Di He, Yaoshuai Ma, Ganzhao Yuan, Yongxiang Liu
More like this →