← Back to all news
arXiv Machine Learning By Ruoran Xu

Blog: Survey of Optimizers

Read the original on arXiv Machine Learning →

The article surveys recent neural‑network optimizers, noting that the field has moved beyond simple Adam variants to encompass matrix‑ and layer‑level designs, time‑policy horizons, and state representations that survive sharding and low‑precision computation. It categorizes optimizers along four axes—temporal estimation, update geometry, horizon management, and representation & systems—highlighting methods such as Muon, Shampoo, SOAP, and quantized states. The survey concludes that while matrix‑aware methods are a genuine advance, no single optimizer universally replaces AdamW, and performance depends on model scale, data‑to‑parameter ratio, batch size, schedule, partitioning, tuning budget, and target metric.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
6d ago

Fantastic Pretraining Optimizers and Where to Find Them

The paper evaluates ten deep learning optimizers across models ranging from 0.1B to 1.2B parameters and varying data-to-model ratios, highlighting that fair comparisons require rigorous hyperparameter tuning and end-of-training evaluations. It finds that claimed speedups over AdamW are lower than advertised, dropping to only 1.1× for the largest models, and that intermediate checkpoint comparisons can mislead due to learning‑rate decay. The study shows that matrix‑based optimizers like Muon and Soap achieve the highest speedups, but these benefits diminish as model size increases.

By Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang
llms
More like this →
arXiv Machine Learning
Sep 7

Optimizer Memory Schedules for Outscaling the Overtraining Axis

The paper studies how different optimizers perform as training duration (overtraining) increases, focusing on matrix‑preconditioned methods (Muon, SOAP) and a momentum‑scheduled method (ADANA) compared to AdamW. Across models ranging from 51M to 253M parameters and overtraining factors up to 256×, the authors find that optimal learning‑rate schedules, weight‑decay coefficients, and memory settings shift with horizon, and that ADANA consistently outperforms AdamW, especially with log‑time weight decay and momentum cooldown. Muon and SOAP maintain roughly constant token‑efficiency advantages, with SOAP potentially improving at the highest overtraining levels.

By Katie Everett, Shikai Qiu
More like this →
arXiv AI
Jul 24

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

arXiv:2607. 20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.

By Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, Tijmen Blankevoort
llms
More like this →