The paper evaluates Muon, a data‑efficient optimizer, against AdamW on a single A100 GPU across 17.7M‑ and 33.6M‑parameter causal language models and a 2.70M‑parameter vision transformer. Using a 23‑candidate development budget and five matched seeds, Muon consistently reduces validation‑loss area under the curve by 3.25–22.44% and lowers test loss in all 20 primary pairs, while AdamW outperforms Muon on language tasks by 3.38–6.96%. Muon’s full optimizer call consumes 56.1–69.6% of training time versus 6.1–9.1% for AdamW, and it shows higher expected regret under restricted candidate pools, indicating trade‑offs between data efficiency and computational cost for small models with reference kernels.
By Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri, Mohamed N. Nounou
arXiv:2609.24678v1 Announce Type: new
Abstract: Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past...
By Sebastian George Sincari (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Bogdan Alexandru Gheorghe (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Antonio Barbalau (Bitdefender, Bucharest, Romania)
The paper introduces Temperon, a training strategy that uses plain SGD for the first 43% of the epoch budget and then hands off to a SAM‑wrapped Muon refiner for the remaining training. On datasets such as CIFAR‑10/100, SVHN, and Tiny ImageNet, Temperon achieves the same or better accuracy as full‑time SAM while reaching key performance targets faster and at lower cost. Ablation studies show that the Muon refiner contributes the majority of the performance gain, while the initial SGD explorer and its restarts add negligible benefit.
By Stamatis Mastromichalakis
arXiv:2606. 12921v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensitive to initialization choices, its optimal learning rates transfer poorly across ranks, and it often fails to beat dense baselines.
By Franz Louis Cesista, Katherine Crowson, C\'edric Simal, Stella Biderman
arXiv:2505. 23725v3 Announce Type: replace Abstract: DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints.
By Benjamin Th\'erien, Xiaolong Huang, Aaron Defazio, Irina Rish, Eugene Belilovsky
arXiv:2607. 23777v1 Announce Type: cross Abstract: The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data.
By Anuj Apte