arXiv:2610. 13799v1 Announce Type: cross Abstract: Variance reduction improves the convergence rate of stochastic non-convex optimization under the average smoothness condition.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
arXiv:2603. 09923v4 Announce Type: replace Abstract: Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam.
arXiv:2610.07290v1 Announce Type: cross Abstract: Recent advances in penalty-based methods for stochastic bilevel optimization (SBO) have eliminated the need for second-order derivative oracles. Howe...
The paper investigates variance reduction techniques for non‑convex optimization when gradient evaluations are delayed. It introduces an ordered recursive momentum estimator whose oracle complexity depends on smoothness, variance, and delay, and provides bounds for both stochastic and finite‑sum settings. An asynchronous snapshot variant is also proposed, allowing updates during full‑gradient computations under a completion‑time bound.
arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.
arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...
arXiv:2609.30499v1 Announce Type: new Abstract: Uniform noise-moment bounds exclude stochastic gradients whose variability increases with the iterate. We study ordinary, single-sample stochastic grad...