arXiv Machine Learning
Stochastic Optimization Under Power-Law Spectra: Tight Bounds and Shuffling Analysis
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
2d ago
From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes
arXiv:2609.40148v1 Announce Type: new Abstract: Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the...
arXiv Machine Learning
Sep 14
High-Probability Convergence of SGD via Batched Updates
The paper introduces Batched SGD, a variant that groups online samples into epochs and performs a single update per epoch using a low‑variance gradient estimate. This batching approach allows a straightforward high‑probability analysis without restrictive assumptions or auxiliary sequences, yielding near‑optimal rates for both strongly convex and non‑convex objectives under standard smoothness and sub‑Gaussian noise conditions. The authors also extend the method to federated learning, providing the first high‑probability guarantees with logarithmic communication complexity, linear speedup in the number of agents, and robustness to data heterogeneity.
arXiv Machine Learning
Jul 30
Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent
arXiv:2606. 06772v2 Announce Type: replace-cross Abstract: Characterizing the optimization dynamics and statistical performance of over-parameterized deep neural networks (DNNs) remains a central challenge in understanding the remarkable success of deep learning.
arXiv Machine Learning
Sep 10
SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
The paper analyzes training dynamics of multiclass logistic regression on high‑dimensional Gaussian mixture models with many classes. It finds that learning proceeds sequentially from the most to the least frequent classes and, when class priors follow a power‑law, the cross‑entropy risk evolves through an initial plateau, a power‑law decay phase, and a final convergence phase. The study also shows how model capacity and optimization trade‑off under a fixed compute budget, leading to a compute‑optimal scaling law that prescribes model size and training time as functions of compute.
arXiv Machine Learning
Sep 10
Speed Limit for Information Acquisition in Stochastic Learning Dynamics
arXiv:2609.08219v1 Announce Type: cross Abstract: Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stocha...
arXiv Machine Learning
Sep 7
The Sample Complexity of Learning Lipschitz Operators with respect to Gaussian Measures
The paper investigates how many linear samples are needed to learn Lipschitz operators under Gaussian measures. It establishes both lower and upper bounds on the Hermite polynomial approximation error and shows that the minimal worst‑case error cannot converge algebraically with the number of samples. However, if the covariance operator of the Gaussian measure decays rapidly, convergence rates arbitrarily close to any algebraic rate can be achieved.