Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

The paper investigates how momentum methods affect large‑batch training in a one‑pass setting using power‑law kernel regression. It derives critical learning rates for SGD, Polyak, and Nesterov, and shows how these rates depend on batch size, momentum, and model capacity. The authors provide scaling laws for risk dynamics, a three‑regime batch‑size phase diagram, and demonstrate that Polyak increases the critical batch size while Nesterov improves data efficiency in the large‑batch regime.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.