arXiv Machine Learning
Muon Can Outperform Dedicated Continual Learning Methods
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
3d ago
Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size
The study re‑examines scaled weight decay (Muon‑SW) versus constant decoupled weight decay for language models, tuning learning‑rate and weight‑decay jointly across a range of model sizes. It finds that most of the reported performance gain is due to a tuning gap, with the tuned advantage shrinking to only 0.0026 BPB. However, the remaining advantage grows with model size, rising from 0.0036 to 0.0051 BPB over an eight‑fold increase in active parameters, and the optimal LR‑WD product shifts faster for Muon‑SW.
arXiv AI
Jul 24
The Active Ingredient in Muon's Grokking
arXiv:2607. 20512v1 Announce Type: cross Abstract: The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW.
arXiv Machine Learning
Sep 17
Temperon: Full-Time SAM Quality at a Third Less Wall-Clock
The paper introduces Temperon, a training strategy that uses plain SGD for the first 43% of the epoch budget and then hands off to a SAM‑wrapped Muon refiner for the remaining training. On datasets such as CIFAR‑10/100, SVHN, and Tiny ImageNet, Temperon achieves the same or better accuracy as full‑time SAM while reaching key performance targets faster and at lower cost. Ablation studies show that the Muon refiner contributes the majority of the performance gain, while the initial SGD explorer and its restarts add negligible benefit.
arXiv Machine Learning
Aug 31
When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
arXiv:2608.27518v1 Announce Type: new Abstract: Continual learning (CL) and model merging (MM) both aim to obtain a single model that performs well across multiple tasks, challenged respectively by c...
arXiv Machine Learning
Jun 3
MuLoCo: Muon is a practical inner optimizer for DiLoCo
arXiv:2505. 23725v3 Announce Type: replace Abstract: DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints.
arXiv AI
Jun 12
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
arXiv:2606. 12921v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensitive to initialization choices, its optimal learning rates transfer poorly across ranks, and it often fails to beat dense baselines.