← Back to all news
arXiv Machine Learning By Shuchen Zhu, Yuxin Fang, Mingze Wang, Kun Yuan

Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining

Read the original on arXiv Machine Learning →

The paper introduces a curvature‑conditioned multiscale momentum algorithm with sphere constraints to accelerate large‑language‑model pretraining. By applying a slow‑decay component for noise reduction and a fast‑decay component for curvature adaptation only along flat directions, the method improves training dynamics without causing parameter inflation. Experiments demonstrate significant speed‑ups for Muon across various architectures and model sizes, and the authors provide theoretical justification for the observed acceleration.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 28

Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement

The paper introduces LITE, a strategy that accelerates large language model pre‑training by enhancing dynamics along flat directions in the optimization landscape. It builds on a Riemannian ODE framework to interpret adaptive optimizers, then applies larger Hessian damping coefficients and learning rates to improve convergence. Experiments show LITE speeds up Muon and SOAP across various architectures, scales, datasets, and learning‑rate schedules.

By Shuchen Zhu, Rizhen Hu, Mingze Wang, Mou Sun, Xue Wang, Kun Yuan, Zaiwen Wen
llms
More like this →
arXiv Machine Learning
Sep 11

Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.

By Zhuanghua Liu, Menglian Wang, Luo Luo
llms
More like this →
arXiv Machine Learning
Jul 15

AMUSE: Anytime Muon with Stable Gradient Evaluation

arXiv:2605. 22432v2 Announce Type: replace Abstract: Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the update geometry by orthogonalizing momentum for matrix parameters.

By Jueun Kim, Baekrok Shin, Jihun Yun, Beomhan Baek, Minhak Song, Chulhee Yun
llms
More like this →