The paper evaluates ten deep learning optimizers across models ranging from 0.1B to 1.2B parameters and varying data-to-model ratios, highlighting that fair comparisons require rigorous hyperparameter tuning and end-of-training evaluations. It finds that claimed speedups over AdamW are lower than advertised, dropping to only 1.1× for the largest models, and that intermediate checkpoint comparisons can mislead due to learning‑rate decay. The study shows that matrix‑based optimizers like Muon and Soap achieve the highest speedups, but these benefits diminish as model size increases.
By Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang
arXiv:2607. 20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.
By Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, Tijmen Blankevoort
arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.
By Naoki Sato, Hiroki Naganuma, Hideaki Iiduka
arXiv:2608. 04407v1 Announce Type: cross Abstract: Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable.
By Masato Fujitake
The paper introduces a physical response-and-memory model for the Muon optimizer, explaining its semi‑orthogonalized momentum update as the maximally dissipative direction under an output‑side safety budget. It treats the weight matrix as a responsive medium with internal stress, showing that momentum corresponds to accumulated stress whose relaxation occurs over multiple timescales—fast and slow. Based on this, the authors propose the Bi‑Maxwell optimizer, which uses a two‑timescale memory kernel and achieves target loss in fewer steps on a public large‑language‑model benchmark.
By Yinze Hu, Hongjun Xiang, Xingao Gong, Hongyu Yu
The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.
By Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland
arXiv:2606. 13894v1 Announce Type: cross Abstract: AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory.
By Nadav Benedek, Tomer Koren, Ohad Fried
The article surveys recent neural‑network optimizers, noting that the field has moved beyond simple Adam variants to encompass matrix‑ and layer‑level designs, time‑policy horizons, and state representations that survive sharding and low‑precision computation. It categorizes optimizers along four axes—temporal estimation, update geometry, horizon management, and representation & systems—highlighting methods such as Muon, Shampoo, SOAP, and quantized states. The survey concludes that while matrix‑aware methods are a genuine advance, no single optimizer universally replaces AdamW, and performance depends on model scale, data‑to‑parameter ratio, batch size, schedule, partitioning, tuning budget, and target metric.
By Ruoran Xu
Clean is a memory‑efficient optimizer that uses Nyström sketching to approximate SOAP’s curvature preconditioners, reducing memory complexity from quadratic to linear while preserving rich curvature information. It also reintegrates off‑subspace components and offers a low‑precision variant, Q‑Clean, that cuts optimizer memory by over 50% compared to Muon while maintaining strong performance. Clean outperforms AdamW in wall‑clock time and enables pre‑training a 13B‑parameter model on a single 80GB GPU.
By Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi, Shikai Fang, Guillaume Rabusseau, Sirisha Rambhatla
arXiv:2610.04204v2 Announce Type: replace
Abstract: Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature,...
By Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi Shikai Fang, Guillaume Rabusseau, Sirisha Rambhatla
arXiv:2608.24593v1 Announce Type: new
Abstract: Adaptive optimizers retain gradient history in moment variables, allowing a local change in loss weighting to alter later updates. We examine whether t...
By Jinhui Guo
The paper reports on the updated AlgoPerf V1 benchmark, evaluating new training algorithms on wall‑clock time‑to‑target across nine workloads on fixed hardware. Using a self‑tuning ruleset that requires a single configuration for all workloads, Schedule‑Free AdamW achieved the best overall performance among 13 PyTorch and JAX submissions. The study also examines trade‑offs between step efficiency and runtime, finding that matrix‑preconditioned and two‑loop methods are sample‑efficient but need aggressive computational optimizations, and highlights the high sensitivity of learning rate and momentum across domains, underscoring the need for more robust hyperparameter‑tuning‑resistant methods.
By Priya Kasimbeg (Alice), Ahmed Khaled (Alice), Aahlad Chandrabhatta (Alice), Yifan Wang (Alice), Andy Liu (Alice), Niccol\`o Ajroldi (Alice), David Tweedle (Alice), Xingyu (Alice), Yang, Isaac Patole, Frank Schneider, George E. Dahl