The paper investigates how momentum methods affect large‑batch training in a one‑pass setting using power‑law kernel regression. It derives critical learning rates for SGD, Polyak, and Nesterov, and shows how these rates depend on batch size, momentum, and model capacity. The authors provide scaling laws for risk dynamics, a three‑regime batch‑size phase diagram, and demonstrate that Polyak increases the critical batch size while Nesterov improves data efficiency in the large‑batch regime.
arXiv:2606. 19179v1 Announce Type: cross Abstract: Stochastic momentum methods such as heavy ball (HB), Nesterov momentum, and variants of Accelerated SGD (ASGD) [Kidambi et al.
By Depen Morwani, Alexandru Meterez, Pranav Nair, Sham Kakade
arXiv:2609. 40148v1 Announce Type: new Abstract: Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss.
By Yichen Wang, Fanghui Liu, Yudong Chen
The paper studies momentum-based optimization when updates are sparse, focusing on two tractable models: a least‑squares problem with sparse inputs and a logistic regression problem with a rare class. It derives exact closed‑form second‑moment dynamics and analyzes their high‑dimensional limits across three scaling exponents for sparsity, batch size, and momentum decay. The resulting phase diagram is governed by the ratio of a momentum retention timescale and a learning timescale, revealing regimes where the dynamics match SGD, become unstable, or recover classical heavy‑ball behavior, and showing that oscillatory behavior depends on token sparsity, leading to a spectral conflict for global momentum.
By Katie Everett, Elliot Paquette
arXiv:2602.06797v3 Announce Type: replace-cross
Abstract: We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training d...
By Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu
The paper presents empirical scaling laws for autoregressive language models, linking prediction loss to model size, data size, and compute, and investigates their theoretical basis using a teacher–student linear RNN framework. In this tractable setting, a stable latent linear RNN generates trajectories while a sketched linear recurrent student is trained via full‑batch WSD gradient descent on next‑token prediction. The study derives explicit approximation, optimization, and statistical scaling laws that depend on the sketch dimension, number of trajectories, and trajectory length, revealing how different power‑law exponents for innovation and initialization covariances affect the rates and crossovers between regimes.
By Ziyan Chen, Zhongzhu Zhou, Peilin Liu, Ding-Xuan Zhou