← Back to all news
arXiv Machine Learning By Eugene Belilovsky, Stefan Horoi, Edouard Oyallon, Benjamin Th\'erien

How Inner Optimizers Shape Replica Scaling

Read the original on arXiv Machine Learning →

The paper investigates how different inner optimizers affect replica scaling when training large language models across distributed compute resources. It identifies two key factors—covariance of pseudo-gradients and batch-size impact on noise scale—that influence performance as replicas increase. The authors show that whitening-based pre‑conditioning as an inner optimizer yields strong replica scaling, improving performance relative to DP as model size and data grow.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jun 3

MuLoCo: Muon is a practical inner optimizer for DiLoCo

arXiv:2505. 23725v3 Announce Type: replace Abstract: DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints.

By Benjamin Th\'erien, Xiaolong Huang, Aaron Defazio, Irina Rish, Eugene Belilovsky
llmsefficiency
More like this →
arXiv Machine Learning
Aug 27

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.

By Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland
llms
More like this →
arXiv Machine Learning
Oct 8

The Best Optimizer Depends on Batch Size

The paper investigates how the choice of optimizer depends on batch size, challenging the assumption that a single optimizer remains best across different batch sizes. It demonstrates that no single scaling rule for Muon consistently works across training settings, and that the optimal optimizer for language model pretraining varies with batch size even after thorough hyperparameter tuning.

By Xingyu Dang, Kaiyue Wen, Sadhika Malladi
llms
More like this →
arXiv Machine Learning
Aug 31

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

The paper investigates the often-overlooked scale vectors in large language models, showing that despite their tiny size they are crucial for pre‑training performance. The authors provide theoretical insights that scale vectors mainly aid optimization rather than expressivity, and they analyze how weight decay affects different normalization layers. Building on these findings, they propose lightweight improvements—branch‑specific heterogeneity, better placement, and magnitude‑direction reparameterization—that consistently reduce loss across a range of model sizes and training settings.

By Mingze Wang, Shuchen Zhu, Yuxin Fang, Binghui Li, Kai Shen, Shu Zhong
llms
More like this →
arXiv AI
Jul 24

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

arXiv:2607. 20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.

By Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, Tijmen Blankevoort
llms
More like this →