The paper introduces a physical response-and-memory model for the Muon optimizer, explaining its semi‑orthogonalized momentum update as the maximally dissipative direction under an output‑side safety budget. It treats the weight matrix as a responsive medium with internal stress, showing that momentum corresponds to accumulated stress whose relaxation occurs over multiple timescales—fast and slow. Based on this, the authors propose the Bi‑Maxwell optimizer, which uses a two‑timescale memory kernel and achieves target loss in fewer steps on a public large‑language‑model benchmark.
By Yinze Hu, Hongjun Xiang, Xingao Gong, Hongyu Yu
arXiv:2610.31728v1 Announce Type: new
Abstract: Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study...
By Xiaoran Chen, Binghui Li, Liwei Wang, Pinyan Lu
arXiv:2604. 00004v2 Announce Type: replace-cross Abstract: The extension of context windows in Large Language Models is typically facilitated by scaling positional encodings followed by lightweight Continual Pre-Training (CPT).
By Ning Yang, Hengyu Zhong, Wentao Wang, Baoliang Tian, Haijun Zhang, Jun Wang
The paper introduces two variance‑adaptive variants of the Muon optimizer—Muon‑NSR and Muon‑VS—for language model pretraining. Both methods incorporate gradient‑variance information into Muon’s orthogonalization process without adding extra hyperparameters, preserving its spectral normalization structure. Experiments on Llama‑style and GPT‑2 models ranging from 125 M to 1.2 B parameters show that these variants outperform well‑tuned Muon baselines and achieve up to a 1.33× step‑to‑target speedup on Llama‑1.2B.
By Jingru Li, Yibo Fan, Huan Li
arXiv:2606. 03899v1 Announce Type: new Abstract: Muon has recently demonstrated strong empirical performance in large language model training, but the theoretical role of momentum in Muon remains unclear.
By Xianliang Li, Zihan Zhang, Weiyang Liu, Han Bao
arXiv:2610.27674v1 Announce Type: new
Abstract: Momentum is central to stochastic optimization in deep learning, including the training of large language models. Classical momentum methods typically...
By Haifeng Xu, Yijin Ren, Qi Deng, Chenyu Zhou, Dongdong Ge, Yinyu Ye
The paper introduces Self-Regularized Flow Language Models (SERF), a self‑supervised framework that enhances continuous diffusion language models by leveraging the denoising trajectory. SERF employs noisy‑to‑clean self‑distillation, where an exponential moving average teacher provides additional supervision along the same interpolation path, and offers two variants—SERF‑L aligning intermediate representations and SERF‑P aligning output token distributions. Experiments on three continuous language‑model paradigms show significant gains in training speed, generation quality, and sampling efficiency, achieving up to 7.7× faster training, 24.3% MAUVE improvement, and 28.8% higher GSM8K accuracy.
"whyItMatters":"The results demonstrate that noisy‑to‑clean self‑distillation can substantially improve continuous diffusion language models without external pretraining or inference changes, offering a practical path to faster, higher‑quality language generation."
By Theodoros Kouzelis, Spyros Gidaris, Nikos Komodakis
Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.
By Zhuanghua Liu, Menglian Wang, Luo Luo
arXiv:2604. 09967v2 Announce Type: replace-cross Abstract: Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization.
By Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, Zheng Zhang
arXiv:2605. 22432v2 Announce Type: replace Abstract: Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the update geometry by orthogonalizing momentum for matrix parameters.
By Jueun Kim, Baekrok Shin, Jihun Yun, Beomhan Baek, Minhak Song, Chulhee Yun
arXiv:2606. 16429v1 Announce Type: new Abstract: Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models.
By Zhongzhu Zhou, Qingyang Wu, Junxiong Wang, Mayank Mishra, Shuaiwen Leon Song, Ben Athiwaratkun, Chenfeng Xu
arXiv:2606. 30461v1 Announce Type: new Abstract: State space models (SSMs) have emerged as efficient linear-time alternatives to attention for long-sequence modeling.
By Thai-Khanh Nguyen, Ngoc-Bich-Uyen Vo, Thieu N. Vo, Tan M. Nguyen, Cuong Pham