arXiv:2605. 24316v3 Announce Type: replace Abstract: Mini-batching is central to large-scale optimization, yet its role in statistical scaling laws remains limited.
arXiv:2605. 24316v2 Announce Type: replace Abstract: Scaling laws provide compact descriptions of how prediction error varies with compute, model size, and data, but existing theory mainly treats single-sample SGD or full data reuse, leaving the role of mini-batching unclear.
The paper investigates how momentum methods affect large‑batch training in a one‑pass setting using power‑law kernel regression. It derives critical learning rates for SGD, Polyak, and Nesterov, and shows how these rates depend on batch size, momentum, and model capacity. The authors provide scaling laws for risk dynamics, a three‑regime batch‑size phase diagram, and demonstrate that Polyak increases the critical batch size while Nesterov improves data efficiency in the large‑batch regime.
The paper introduces Batched SGD, a variant that groups online samples into epochs and performs a single update per epoch using a low‑variance gradient estimate. This batching approach allows a straightforward high‑probability analysis without restrictive assumptions or auxiliary sequences, yielding near‑optimal rates for both strongly convex and non‑convex objectives under standard smoothness and sub‑Gaussian noise conditions. The authors also extend the method to federated learning, providing the first high‑probability guarantees with logarithmic communication complexity, linear speedup in the number of agents, and robustness to data heterogeneity.
arXiv:2602. 11557v2 Announce Type: replace Abstract: A variety of widely used optimization methods like SignSGD and Muon can be interpreted as instances of steepest descent under different norm-induced geometries.
arXiv:2402.11215v4 Announce Type: replace Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-sc...
arXiv:2508. 03105v3 Announce Type: replace Abstract: We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function.
arXiv:2606. 19179v1 Announce Type: cross Abstract: Stochastic momentum methods such as heavy ball (HB), Nesterov momentum, and variants of Accelerated SGD (ASGD) [Kidambi et al.
arXiv:2607. 27731v1 Announce Type: new Abstract: Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics.
arXiv:2602. 02431v2 Announce Type: replace-cross Abstract: It is folklore that reusing training data more than once can improve the statistical efficiency of gradient-based learning.
arXiv:2607. 01487v1 Announce Type: new Abstract: We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called three-term law).
The paper investigates how momentum methods affect large‑batch training in a one‑pass setting using power‑law kernel regression. It derives critical learning rates for SGD, Polyak, and Nesterov, and shows how these rates depend on batch size, momentum, and a capacity exponent. The authors then analyze risk dynamics, optimize final‑step risk under a fixed data budget, and present a three‑regime batch‑size phase diagram that highlights Polyak’s ability to enlarge the critical batch size and Nesterov’s superior data efficiency in the large‑batch regime.
arXiv:2510. 14717v2 Announce Type: replace-cross Abstract: Increasing the batch size during training -- a ''batch ramp'' -- is a promising strategy to accelerate large language model pretraining.