← Back to all news
arXiv Statistics ML By Nabil Kahal\'e

Stochastic gradient descent with initial regularization

Read the original on arXiv Statistics ML →

The paper studies a variant of stochastic gradient descent called SGDIR, which incorporates initial regularization. It derives dimension‑free upper bounds on the expected excess risk for the squared loss, providing new rates for both averaged and non‑averaged SGDIR under various assumptions. The authors also establish matching lower bounds in certain regimes and compare SGDIR to ridge regression in noisy settings, showing comparable performance up to a polylogarithmic factor.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Statistics ML.

Related stories

arXiv Machine Learning
Jun 10

Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization

arXiv:2509. 17251v2 Announce Type: replace-cross Abstract: Existing theory suggests that for linear regression problems categorized by capacity and source conditions, gradient descent (GD) is always minimax optimal, while both ridge regression and online stochastic gradient descent (SGD) are polynomially suboptimal for certain categories of such problems.

By Jingfeng Wu, Peter L. Bartlett, Sham M. Kakade, Jason D. Lee, Bin Yu
More like this →
arXiv Machine Learning
Oct 23

The Mathematical Theory of Double Descent as Shrinkage, not Overfitting: Matching Bounds for Path-Regularized Multilayer Neural Networks

The paper presents a nonasymptotic generalization theory for multilayer fully connected neural networks trained with path regularization, covering both underparameterized and overparameterized regimes without assuming bounded loss, sparsity, or width/depth limits. It explicitly models approximation error, resolves an open problem on approximation rates in generalized Barron spaces, and establishes matching upper and lower bounds that show the error curve is non‑monotone—descending, rising, then descending again—as width increases. The analysis reveals that the rise is due to shrinkage imposed by regularization rather than overfitting, and this effect cannot be eliminated by width‑independent regularization.

By Hao Yu
More like this →