arXiv Machine Learning
Jun 8
Flatland: The Adventures of Gradient Descent with Large Step Sizes
arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.
Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes
arXiv:2406. 05033v3 Announce Type: replace Abstract: We study gradient descent (GD) dynamics on logistic regression problems with large, constant step sizes.
Related stories
arXiv Statistics ML
2d ago
Tight Transition Time Bounds for Separable Logistic Regression at the Edge of Stability
arXiv:2610.01459v1 Announce Type: cross Abstract: We study logistic regression on linearly separable data under gradient descent with a large constant stepsize $\eta$. Such dynamics may exhibit a cha...
arXiv Machine Learning
Jun 16
Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimension
arXiv:2602. 12471v2 Announce Type: replace Abstract: We consider the optimization problem of minimizing the logistic loss with gradient descent to train a linear model for binary classification with separable data.
arXiv Machine Learning
Jun 18
On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials
arXiv:2602. 14789v2 Announce Type: replace Abstract: The dynamical stability of the iterates during training plays a key role in determining the minima obtained by optimization algorithms.
arXiv Machine Learning
Sep 2
The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow
arXiv:2609.01034v1 Announce Type: new Abstract: The central flow of Cohen et al. (2025) is an empirically accurate continuous-time model of gradient descent at the edge of stability in deep learning,...
arXiv Machine Learning
Jun 2
Accelerating Min-Max Optimization via Power-Law Stepsizes
arXiv:2606. 01764v1 Announce Type: cross Abstract: We revisit the convergence guarantees of the Extragradient (EG) method for unconstrained biaffine min-max optimization.
arXiv Machine Learning
Aug 12
A lower bound for stepsize-based acceleration of gradient descent
arXiv:2608. 10418v1 Announce Type: cross Abstract: Recent work has shown that, for smooth convex optimization, plain gradient descent can be accelerated from its textbook convergence rate of $O(T^{-1})$ (where $T$ denotes the number of iterations) to $O\big(T^{-\log_2(1+\sqrt{2})}\big)$ using carefully designed stepsize schedules alone, without resorting to momentum or other algorithmic modifications.
arXiv Machine Learning
Jun 4
Pseudospectral Bounds for Transient Amplification in Coupled Gradient Descent
arXiv:2606. 04031v1 Announce Type: new Abstract: Coupled gradient descent--where the update of one parameter block depends on another--underlies bilevel optimization, two-time-scale stochastic approximation, and adversarial training.
arXiv Machine Learning
Jul 21
Scaling Limits of Constant-Stepsize SGD at Flat Minima
arXiv:2607. 16384v1 Announce Type: new Abstract: For stochastic gradient descent (SGD) with a constant stepsize $\alpha$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons.
Hugging Face Trending Papers
Sep 3
Projected Riemannian Gradient Descent for the Bures-Wasserstein Barycenter: Dimension-Independent Linear Convergence at Unit Step Size
The paper introduces a Projected Riemannian Gradient Descent (RGD) algorithm for computing the Bures‑Wasserstein barycenter of positive definite matrices, achieving dimension‑independent linear convergence at unit step size. It resolves a previous dichotomy by showing that clipping eigenvalues to a fixed interval yields a closed‑form, non‑expansive projection in the BW metric, allowing the algorithm to match the empirical speed of unit‑step RGD while maintaining theoretical guarantees. The method also extends to the invariant matrix projection problem, providing a unified dimension‑independent analysis.
arXiv Machine Learning
Sep 16
Information Geometric Self-Organization at the Edge of Stability in High-Capacity Kernel Associative Memories
arXiv:2609.16827v1 Announce Type: new Abstract: High-capacity associative memories based on Kernel Logistic Regression (KLR) exhibit exceptional storage capabilities and robustness. Previous empirica...
arXiv Machine Learning
Jul 2
Zeroth-Order Optimization at the Edge of Stability
arXiv:2604. 14669v2 Announce Type: replace Abstract: Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models, yet their optimization dynamics in deep learning remain underexplored.