← Back to all news
arXiv Machine Learning By Shuchen Zhu, Yuxin Fang, Mingze Wang, Kun Yuan

Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining

Read the original on arXiv Machine Learning →

The paper introduces a curvature‑conditioned multiscale momentum algorithm with sphere constraints to accelerate large‑language‑model pretraining. By applying a slow‑decay component for noise reduction and a fast‑decay component for curvature adaptation only along flat directions, the method improves training dynamics without causing parameter inflation. Experiments demonstrate significant speed‑ups for Muon across various architectures and model sizes, and the authors provide theoretical justification for the observed acceleration.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 11

Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

Musec introduces MomentUm SpEctral Clipping, an optimizer-level, architecture‑agnostic technique that replaces Muon’s spectral flattening with selective spectral clipping to stabilize training. By clipping singular values above a threshold while preserving the momentum’s spectral structure, Musec addresses loss spikes and unbounded weight growth without requiring architecture‑specific changes. Soft Musec, an efficient implementation using smooth spectral saturation via coupled Newton‑Schulz iterations, offers convergence guarantees in nonconvex nonsmooth stochastic optimization and empirically improves stability across diverse learning rates and model sizes.

By Zhuanghua Liu, Menglian Wang, Luo Luo
llms
More like this →
arXiv Machine Learning
Jul 15

AMUSE: Anytime Muon with Stable Gradient Evaluation

arXiv:2605. 22432v2 Announce Type: replace Abstract: Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the update geometry by orthogonalizing momentum for matrix parameters.

By Jueun Kim, Baekrok Shin, Jihun Yun, Beomhan Baek, Minhak Song, Chulhee Yun
llms
More like this →
arXiv Machine Learning
Jul 27

Hyperball May Not Be a Free Lunch

arXiv:2607. 22444v1 Announce Type: new Abstract: For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates.

By Yihao Xiao, Jialong Sun, Zitian Gao, Zeming Wei, Chutian Wang, Ran Tao, Jiaye Teng, Bryan Dai
More like this →