← Back to all news
arXiv Machine Learning By Miao Lu, Zeyu Bian, Kaiyue Wen, Beining Wu, Siyu Chen, Tianhao Wang, Zhiyuan Li

Towards Understanding Momentum Acceleration in River-Valley Loss Landscape

Read the original on arXiv Machine Learning →

The paper investigates how momentum influences optimization in a river‑valley loss landscape, where a low‑loss manifold is surrounded by steep orthogonal directions. It shows that momentum stabilizes large learning rates that vanilla gradient descent cannot tolerate, enabling faster progress along the river. The study also finds that in very flat, slowly spinning rivers, momentum itself does not directly accelerate tracking, but the larger permissible learning rate does.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jul 15

AMUSE: Anytime Muon with Stable Gradient Evaluation

arXiv:2605. 22432v2 Announce Type: replace Abstract: Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the update geometry by orthogonalizing momentum for matrix parameters.

By Jueun Kim, Baekrok Shin, Jihun Yun, Beomhan Baek, Minhak Song, Chulhee Yun
llms
More like this →
arXiv AI
3d ago

How Does Local Landscape Geometry Evolve in Language Model Pre-Training?

The paper investigates how the local landscape geometry of language model pre‑training evolves, identifying two distinct phases. In Phase I, the landscape starts sharp, causing instability and loss plateaus at high learning rates, which explains the need for learning‑rate warmup and suggests longer warmups for larger peak rates. In Phase II, the geometry is governed by gradient noise scale, revealing a depth‑flatness trade‑off that motivates a dynamic batch‑size scheduler that starts small and grows later in training.

By Zhanpeng Zhou, Yuhan Sun, Bingrui Li, Jinbo Wang, Huaijin Wu, Lei Wu, Junchi Yan
llms
More like this →