Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.
By Zhuanghua Liu, Menglian Wang, Luo Luo
arXiv:2606. 00371v1 Announce Type: new Abstract: Muon optimizers improve neural-network training by replacing ill-conditioned momentum updates with approximately semi-orthogonal updates.
By Hua Huang
The paper introduces DP-Muon, a differentially private optimization method that incorporates matrix‑orthogonalized momentum. It employs standard global per‑example clipping and releases a single Gaussian‑noised gradient per step, with matrix and auxiliary updates treated as post‑processing. The authors analyze the mean distortion introduced when fresh Gaussian noise passes through a nonlinear matrix map, deriving exact Gaussian heat identities and showing that for a smooth Newton‑Schulz map, the conditional output bias is reduced from second to fourth order in the noise scale. They also establish matrix‑block stationarity bounds, quantify orthogonalization error, and provide criteria for improving the upper bound, while a separate inequality captures the impact of auxiliary Adam updates. Experiments on GPT‑2 at various privacy targets demonstrate that DP‑Muon configurations outperform Adam baselines in test negative log‑likelihood.
By Jihwan Kim, Chenglin Fan
arXiv:2607. 14536v1 Announce Type: new Abstract: Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization.
By Da Chang, Qiankun Shi, Lvgang Zhang, Di He, Yaoshuai Ma, Ganzhao Yuan, Yongxiang Liu
arXiv:2604. 09967v2 Announce Type: replace-cross Abstract: Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization.
By Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, Zheng Zhang
arXiv:2512. 04632v2 Announce Type: replace Abstract: Orthogonality-based optimizers, such as Muon, have recently shown strong performance across large-scale training and community-driven efficiency challenges.
By Thibaut Boissin (IRIT-MISFIT), Thomas Massena (DTIPG - SNCF, IRIT-MISFIT), Franck Mamalet (IRIT-MISFIT), Mathieu Serrurier (IRIT-MISFIT)