arXiv:2505. 23725v3 Announce Type: replace Abstract: DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints.
By Benjamin Th\'erien, Xiaolong Huang, Aaron Defazio, Irina Rish, Eugene Belilovsky
The study investigates whether switching optimizers between pretraining and subsequent training stages of large language models (LLMs) incurs a performance penalty. By pretraining 1.2B Llama‑3 models with varying proportions of AdamW and Muon updates, and interpolating between the two during continued pretraining and supervised fine‑tuning (SFT), the authors find little evidence of a meaningful optimizer mismatch penalty. The results show that final performance is largely determined by the independent effects of the pretraining and post‑training optimizers, with downstream performance generally improving as the proportion of Muon updates during SFT increases, and that Muon remains competitive with AdamW even for task‑general SFT on larger models.
By Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma, Rameswar Panda, Paul Pu Liang
arXiv:2607. 01984v1 Announce Type: cross Abstract: Newer lightweight convolutional neural networks are often presented as improving predictive performance and deployment efficiency, but such claims require controlled evaluation.
By Tasnim Shahriar
The paper introduces Temperon, a training strategy that uses plain SGD for the first 43% of the epoch budget and then hands off to a SAM‑wrapped Muon refiner for the remaining training. On datasets such as CIFAR‑10/100, SVHN, and Tiny ImageNet, Temperon achieves the same or better accuracy as full‑time SAM while reaching key performance targets faster and at lower cost. Ablation studies show that the Muon refiner contributes the majority of the performance gain, while the initial SGD explorer and its restarts add negligible benefit.
By Stamatis Mastromichalakis
The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.
By Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland
arXiv:2609.24678v1 Announce Type: new
Abstract: Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past...
By Sebastian George Sincari (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Bogdan Alexandru Gheorghe (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Antonio Barbalau (Bitdefender, Bucharest, Romania)