arXiv Machine Learning
Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
4d ago
Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap
The paper investigates how the optimizer Muon compares to AdamW during fine‑tuning of large language models. While Muon speeds up pretraining loss minimization, the study finds that when pretraining data is sufficiently large, AdamW actually outperforms Muon after fine‑tuning. The authors explain this reversal by introducing the margin‑to‑spread ratio, showing that Muon has a higher ratio at small data budgets but AdamW surpasses it as data grows, a trend confirmed both analytically and in simulations.
arXiv Machine Learning
Sep 2
Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining
The paper introduces two variance‑adaptive variants of the Muon optimizer—Muon‑NSR and Muon‑VS—for language model pretraining. Both methods incorporate gradient‑variance information into Muon’s orthogonalization process without adding extra hyperparameters, preserving its spectral normalization structure. Experiments on Llama‑style and GPT‑2 models ranging from 125 M to 1.2 B parameters show that these variants outperform well‑tuned Muon baselines and achieve up to a 1.33× step‑to‑target speedup on Llama‑1.2B.
arXiv Machine Learning
Aug 27
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.
arXiv Machine Learning
Jun 15
Beyond a Single Explanation of the Adam--SGD Gap
arXiv:2606. 14259v1 Announce Type: new Abstract: Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties.
arXiv AI
Jun 9
Muon Learns More Robust and Transferable Features than Adam
arXiv:2606. 09658v1 Announce Type: cross Abstract: Muon has recently emerged as a state-of-the-art optimizer for pretraining Large Language Models (LLMs) and vision classifiers.
arXiv Machine Learning
Oct 27
When and How Does Muon Precondition? The Role of Structured Gradient Noise
arXiv:2610.31728v1 Announce Type: new Abstract: Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study...