arXiv Machine Learning

Online simultaneous inference for quantiles via smoothed stochastic gradient descent

arXiv Machine Learning
5d ago

Stochastic Gradient Descent: A Nonlinear Time Series Perspective

This paper studies the statistical properties of Stochastic Gradient Descent (SGD) with a constant step size using the framework of iterated random functions. It demonstrates convergence in Euclidean distance via Geometric Moment Contraction, establishes quenched central limit theorems, invariance principles, and α-stable convergence for averaged SGD regardless of initialization, and explores the effect of step sizes through derivative processes. The authors further develop a Richardson-Romberg extrapolation with multiple step sizes, propose a bias-reduced variant, and introduce an online inference method that yields empirical confidence intervals with coverage approaching nominal levels as sample size increases.

By Jiaqi Li, Zhipeng Lou, Stefan Richter, Wei Biao Wu
arXiv Machine Learning
Sep 14

High-Probability Convergence of SGD via Batched Updates

The paper introduces Batched SGD, a variant that groups online samples into epochs and performs a single update per epoch using a low‑variance gradient estimate. This batching approach allows a straightforward high‑probability analysis without restrictive assumptions or auxiliary sequences, yielding near‑optimal rates for both strongly convex and non‑convex objectives under standard smoothness and sub‑Gaussian noise conditions. The authors also extend the method to federated learning, providing the first high‑probability guarantees with logarithmic communication complexity, linear speedup in the number of agents, and robustness to data heterogeneity.

By Feng Zhu, Robert W. Heath Jr., Aritra Mitra