The paper introduces LITE, a strategy that accelerates large language model pre‑training by enhancing dynamics along flat directions in the optimization landscape. It builds on a Riemannian ODE framework to interpret adaptive optimizers, then applies larger Hessian damping coefficients and learning rates to improve convergence. Experiments show LITE speeds up Muon and SOAP across various architectures, scales, datasets, and learning‑rate schedules.
arXiv:2610.04204v2 Announce Type: replace Abstract: Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature,...
Clean is a memory‑efficient optimizer that uses Nyström sketching to approximate SOAP’s curvature preconditioners, reducing memory complexity from quadratic to linear while preserving rich curvature information. It also reintegrates off‑subspace components and offers a low‑precision variant, Q‑Clean, that cuts optimizer memory by over 50% compared to Muon while maintaining strong performance. Clean outperforms AdamW in wall‑clock time and enables pre‑training a 13B‑parameter model on a single 80GB GPU.
MONA is a new optimizer that extends the Muon optimizer by adding a Nesterov‑style acceleration term derived from an exponential moving average of gradient differences. The paper provides a convergence analysis showing that this term offers curvature‑aware corrections while maintaining Muon’s spectral‑norm regularization. Empirical results demonstrate that MONA outperforms both Muon and AdamW on Mixture‑of‑Experts pretraining across models ranging from 1 B to 68 B parameters, and achieves state‑of‑the‑art performance on downstream benchmarks after fine‑tuning the largest model.
The paper introduces a curvature‑conditioned multiscale momentum algorithm with sphere constraints to accelerate large‑language‑model pretraining. By applying a slow‑decay component for noise reduction and a fast‑decay component for curvature adaptation only along flat directions, the method improves training dynamics without causing parameter inflation. Experiments demonstrate significant speed‑ups for Muon across various architectures and model sizes, and the authors provide theoretical justification for the observed acceleration.
arXiv:2605. 22432v2 Announce Type: replace Abstract: Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the update geometry by orthogonalizing momentum for matrix parameters.
arXiv:2607. 20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.
arXiv:2608. 05088v1 Announce Type: new Abstract: Muon has recently emerged as a promising alternative to AdamW for language model pretraining by orthogonalizing momentum matrices using Newton-Schulz iterations.
arXiv:2609.36692v1 Announce Type: cross Abstract: Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram represen...
arXiv:2606. 04662v1 Announce Type: cross Abstract: Muon improves training efficiency over Adam in large language-model training by about two times, but the local geometric source of this advantage remains unclear.
arXiv:2509. 08765v4 Announce Type: replace-cross Abstract: Data-driven acceleration of scientific computing workflows has been a high-profile aim of machine learning (ML) for science, with numerical simulation of transient partial differential equations (PDEs) being one of the main applications.
arXiv:2610.08049v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) is widely used as a parameter-efficient fine-tuning technique for pre-trained deep neural networks, which approximates the w...
arXiv:2610.27674v1 Announce Type: new Abstract: Momentum is central to stochastic optimization in deep learning, including the training of large language models. Classical momentum methods typically...