← Back to all news
Hugging Face Trending Papers

Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

Read the original on Hugging Face Trending Papers →

The paper investigates how momentum methods affect large‑batch training in a one‑pass setting using power‑law kernel regression. It derives critical learning rates for SGD, Polyak, and Nesterov, and shows how these rates depend on batch size, momentum, and model capacity. The authors provide scaling laws for risk dynamics, a three‑regime batch‑size phase diagram, and demonstrate that Polyak increases the critical batch size while Nesterov improves data efficiency in the large‑batch regime.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

Related stories

arXiv Machine Learning
Sep 3

Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

The paper investigates how momentum methods affect large‑batch training in a one‑pass setting using power‑law kernel regression. It derives critical learning rates for SGD, Polyak, and Nesterov, and shows how these rates depend on batch size, momentum, and a capacity exponent. The authors then analyze risk dynamics, optimize final‑step risk under a fixed data budget, and present a three‑regime batch‑size phase diagram that highlights Polyak’s ability to enlarge the critical batch size and Nesterov’s superior data efficiency in the large‑batch regime.

By Jia-Nan Wang, Zixun Huang, Kairui Li, Lei Wu
More like this →
arXiv Machine Learning
Oct 19

Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions

The paper studies momentum-based optimization when updates are sparse, focusing on two tractable models: a least‑squares problem with sparse inputs and a logistic regression problem with a rare class. It derives exact closed‑form second‑moment dynamics and analyzes their high‑dimensional limits across three scaling exponents for sparsity, batch size, and momentum decay. The resulting phase diagram is governed by the ratio of a momentum retention timescale and a learning timescale, revealing regimes where the dynamics match SGD, become unstable, or recover classical heavy‑ball behavior, and showing that oscillatory behavior depends on token sparsity, leading to a spectral conflict for global momentum.

By Katie Everett, Elliot Paquette
efficiency
More like this →