arXiv:2610. 13799v1 Announce Type: cross Abstract: Variance reduction improves the convergence rate of stochastic non-convex optimization under the average smoothness condition.
By Wei Jiang, Yibo Wang, Zechao Li
arXiv:2609. 12785v1 Announce Type: new Abstract: Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice.
By Misbah Uz Zaman, Anirbit Mukherjee
The paper investigates variance reduction techniques for non‑convex optimization when gradient evaluations are delayed. It introduces an ordered recursive momentum estimator whose oracle complexity depends on smoothness, variance, and delay, and provides bounds for both stochastic and finite‑sum settings. An asynchronous snapshot variant is also proposed, allowing updates during full‑gradient computations under a completion‑time bound.
By Wei Jiang, Dingzhi Yu, Zechao Li
The paper introduces RanSOM, a second‑order momentum algorithm that eliminates the need for a Lipschitz Hessian or auxiliary forward/backward passes by randomizing the step length. By sampling the step from an exponential distribution and using a Stein‑type identity, the method obtains an unbiased estimate of the gradient change with a single Hessian‑vector product that also serves the next gradient computation. RanSOM-E achieves optimal oracle‑complexity exponents for heavy‑tailed noise without clipping, while RanSOM-B preserves feasibility for Frank‑Wolfe iterates, and experiments on CIFAR‑10 show comparable accuracy to existing methods with lower update time.
By El Mahdi Chayti, Mohamad Fakhouri
arXiv:2610. 33381v2 Announce Type: new Abstract: Training is often stopped at a data-dependent time, so guarantees for the last iterate must hold at all times simultaneously.
By Yuxing Peng, Zhiqing Tang, Weijia Jia
arXiv:2610. 13789v1 Announce Type: new Abstract: Momentum methods such as STORM and second-order momentum track the gradient by adding a transport correction to momentum, at the cost of an extra oracle call per iteration.
By El Mahdi Chayti, Martin Jaggi