← Back to all news
arXiv Machine Learning By Zhou Fan, Leda Wang

High-dimensional learning dynamics of multi-pass Stochastic Gradient Descent in multi-index models

Read the original on arXiv Machine Learning →

The paper presents an asymptotically exact description of the learning dynamics of multi‑pass stochastic gradient descent (SGD) in high‑dimensional non‑convex multi‑index models. By considering sub‑linear batch sizes and a critical learning‑rate scaling, the authors derive a continuous‑time scaling limit governed by dynamical mean‑field equations driven by a scalar Poisson jump process. This framework unifies several existing theories of SGD—gradient flow on empirical risk, autonomous ODEs for online SGD, and diffusion approximations in least‑squares—while revealing distinct behavior for general multi‑pass SGD beyond these special cases.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jul 14

Exact Dynamics of Multi-class Stochastic Gradient Descent

arXiv:2510. 14074v2 Announce Type: replace-cross Abstract: We develop a framework for analyzing the learning dynamics of high-dimensional problems trained using one-pass stochastic gradient descent (SGD) with data from multiple anisotropic classes.

By Elizabeth Collins-Woodfin, Inbar Seroussi
More like this →
arXiv Machine Learning
Oct 16

Stochastic Gradient Descent: A Nonlinear Time Series Perspective

This paper studies the statistical properties of Stochastic Gradient Descent (SGD) with a constant step size using the framework of iterated random functions. It demonstrates convergence in Euclidean distance via Geometric Moment Contraction, establishes quenched central limit theorems, invariance principles, and α-stable convergence for averaged SGD regardless of initialization, and explores the effect of step sizes through derivative processes. The authors further develop a Richardson-Romberg extrapolation with multiple step sizes, propose a bias-reduced variant, and introduce an online inference method that yields empirical confidence intervals with coverage approaching nominal levels as sample size increases.

By Jiaqi Li, Zhipeng Lou, Stefan Richter, Wei Biao Wu
safety
More like this →