← Back to all news
arXiv Machine Learning

Derivative-Free Structured Updates for Muon

The paper introduces a derivative‑free framework for Muon‑style updates, replacing gradient‑based momentum with structured finite differences. Four variants—full entrywise recovery, random low‑rank surrogates, basis‑aligned rank‑one probing, and direct structured search—are explored, with basis‑aligned probing shown to be equivalent to coordinate finite differences up to scaling. Experiments on matrix regression, noisy‑gradient regression, a neural network, and a CartPole task demonstrate that random rank‑one probing can significantly reduce function evaluations, though at the expense of update accuracy, and that accurate function values can sometimes offset unreliable gradient oracles.

Related stories

arXiv AI
Jul 16

Reassessing Muon for Matrix Factorization

arXiv:2607. 13246v1 Announce Type: cross Abstract: Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training.

By Ali Parviz, Gal Mishne, Alex Cloninger
llmsbenchmarks
More like this →
Hugging Face Trending Papers
Jul 14

Reassessing Muon for Matrix Factorization

Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm.

llmsbenchmarks
More like this →
arXiv Machine Learning
Sep 18

Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training

The paper introduces low‑rank orthogonalization, a technique that exploits the low‑rank nature of gradients in neural network training to perform matrix orthogonalization more efficiently. Building on this, the authors present low‑rank matrix‑signed gradient descent (MSGD) and a low‑rank variant of the Muon optimizer, showing through experiments that low‑rank Muon matches or surpasses vanilla Muon on GPT‑2 and LLaMA pretraining, especially for larger models. Theoretical analysis provides iteration‑complexity bounds for both low‑rank MSGD and low‑rank Muon under heavy‑tailed noise.

By Chuan He, Zhanwang Deng, Zhaosong Lu
llms
More like this →
arXiv AI
Jul 2

Muon as a Residual Connection

arXiv:2607. 01124v1 Announce Type: cross Abstract: Muon has recently emerged as one of the most effective optimizers for training large neural networks, yet its empirical success has been explained from several different perspectives.

By Hao Huang
More like this →
arXiv Machine Learning
Jun 9

Convergence Bound and Critical Batch Size of Muon Optimizer

arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.

By Naoki Sato, Hiroki Naganuma, Hideaki Iiduka
computer-vision
More like this →
arXiv Machine Learning
Sep 2

Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining

The paper introduces two variance‑adaptive variants of the Muon optimizer—Muon‑NSR and Muon‑VS—for language model pretraining. Both methods incorporate gradient‑variance information into Muon’s orthogonalization process without adding extra hyperparameters, preserving its spectral normalization structure. Experiments on Llama‑style and GPT‑2 models ranging from 125 M to 1.2 B parameters show that these variants outperform well‑tuned Muon baselines and achieve up to a 1.33× step‑to‑target speedup on Llama‑1.2B.

By Jingru Li, Yibo Fan, Huan Li
llms
More like this →