← Back to all news
arXiv Machine Learning By Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma, Rameswar Panda, Paul Pu Liang

The Ephemeral Cost of Switching Optimizers Across LLM Training Stages

Read the original on arXiv Machine Learning →

The study investigates whether switching optimizers between pretraining and subsequent training stages of large language models (LLMs) incurs a performance penalty. By pretraining 1.2B Llama‑3 models with varying proportions of AdamW and Muon updates, and interpolating between the two during continued pretraining and supervised fine‑tuning (SFT), the authors find little evidence of a meaningful optimizer mismatch penalty. The results show that final performance is largely determined by the independent effects of the pretraining and post‑training optimizers, with downstream performance generally improving as the proportion of Muon updates during SFT increases, and that Muon remains competitive with AdamW even for task‑general SFT on larger models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
4d ago

When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

The paper evaluates Muon, a data‑efficient optimizer, against AdamW on a single A100 GPU across 17.7M‑ and 33.6M‑parameter causal language models and a 2.70M‑parameter vision transformer. Using a 23‑candidate development budget and five matched seeds, Muon consistently reduces validation‑loss area under the curve by 3.25–22.44% and lowers test loss in all 20 primary pairs, while AdamW outperforms Muon on language tasks by 3.38–6.96%. Muon’s full optimizer call consumes 56.1–69.6% of training time versus 6.1–9.1% for AdamW, and it shows higher expected regret under restricted candidate pools, indicating trade‑offs between data efficiency and computational cost for small models with reference kernels.

By Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri, Mohamed N. Nounou
llmsbenchmarks
More like this →
arXiv Machine Learning
Oct 5

The Conflict Between Logic and Memory: Training Conditions for Optimizer-Dependent Rule Acquisition

The paper investigates how different optimizers acquire distinct generalizing relations in single‑hidden‑layer ReLU networks. By manipulating nuisance‑connected weights and performing a strict‑parity sweep, the authors show that training conditions reshape both shared failures and relative optimizer advantages, with specific interventions improving accuracy on hard stages. The study also compares optimizer behavior across tasks and an external architecture (CIFAR‑10).

By Gongyue Zhang, Honghai Liu
More like this →
arXiv Machine Learning
Sep 2

Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining

The paper introduces two variance‑adaptive variants of the Muon optimizer—Muon‑NSR and Muon‑VS—for language model pretraining. Both methods incorporate gradient‑variance information into Muon’s orthogonalization process without adding extra hyperparameters, preserving its spectral normalization structure. Experiments on Llama‑style and GPT‑2 models ranging from 125 M to 1.2 B parameters show that these variants outperform well‑tuned Muon baselines and achieve up to a 1.33× step‑to‑target speedup on Llama‑1.2B.

By Jingru Li, Yibo Fan, Huan Li
llms
More like this →