arXiv Machine Learning
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 2
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
arXiv:2609.39595v1 Announce Type: new Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...
arXiv Machine Learning
Sep 16
Nonsmooth Optimization via Orthogonalized Momentum
arXiv:2609.13677v1 Announce Type: cross Abstract: Modern real application problems involve matrix-valued parameters, yet conventional optimizers treat them as vectors, thereby motivating matrix-aware...
arXiv Machine Learning
Sep 11
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.
arXiv Machine Learning
Jun 3
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
arXiv:2606. 03899v1 Announce Type: new Abstract: Muon has recently demonstrated strong empirical performance in large language model training, but the theoretical role of momentum in Muon remains unclear.
arXiv Machine Learning
Aug 14
Federated Compositional Muon Optimizer for Matrix-Wise Models
arXiv:2608. 12710v1 Announce Type: new Abstract: Muon, a more recently developed optimizer, is useful for matrix-wise models in AI areas.
arXiv AI
Aug 25
A Physical Response-and-Memory Model for Muon Optimization
The paper introduces a physical response-and-memory model for the Muon optimizer, explaining its semi‑orthogonalized momentum update as the maximally dissipative direction under an output‑side safety budget. It treats the weight matrix as a responsive medium with internal stress, showing that momentum corresponds to accumulated stress whose relaxation occurs over multiple timescales—fast and slow. Based on this, the authors propose the Bi‑Maxwell optimizer, which uses a two‑timescale memory kernel and achieves target loss in fewer steps on a public large‑language‑model benchmark.