The paper introduces a parallel architecture for stochastic gradient methods that adaptively selects the number of iterations. An algorithm A(x₀, y) takes an initial point and a step limit y, and p processors search for an appropriate iteration count T using a prescribed function h. The framework guarantees a (p, αₚ)-approximation, meaning for any T ≥ T₀ there exists a processor and stage where the cumulative iterations lie within a factor αₚ of T, and the authors prove tight lower bounds for αₚ while presenting simple arithmetic stochastic gradient methods that use only divisions by powers of two.
By Bin Fu
arXiv:2502.21099v3 Announce Type: replace-cross
Abstract: This paper proposes {\sf AEPG-SPIDER}, an Adaptive Extrapolated Proximal Gradient (AEPG) method with variance reduction for minimizing compos...
By Ganzhao Yuan
arXiv:2505.20817v3 Announce Type: replace-cross
Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...
By Taha El Bakkali El Kadi, Savelii Chezhegov, Aleksandr Beznosikov, Samuel Horv\'ath, Eduard Gorbunov
arXiv:2603. 10485v3 Announce Type: replace-cross Abstract: In this work, we study the convergence properties of the Dual Space Preconditioned Gradient Descent, encompassing optimizers such as Normalized Gradient Descent and Gradient Clipping.
By Reza Ghane, Danil Akhtiamov, Babak Hassibi
arXiv:2605. 18694v2 Announce Type: replace-cross Abstract: Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process.
By Zijian Liu
arXiv:2609.15723v1 Announce Type: new
Abstract: Traditional variance reduction methods (e.g., SPIDER, SARAH, STORM) have been extensively investigated for improving the convergence rates of stochasti...
By Wei Jiang, Sifan Yang, Yibo Wang, Lijun Zhang, Zechao Li
arXiv:2606. 01764v1 Announce Type: cross Abstract: We revisit the convergence guarantees of the Extragradient (EG) method for unconstrained biaffine min-max optimization.
By Yue Wu, Weiqiang Zheng, Yang Cai, Haipeng Luo
While entropy regularization is widely used to stabilize and accelerate Natural Policy Gradient methods, its ability to yield faster convergence rates for the unregularized objective remains underexplored. Existing analyses often rely on double-loop architectures and invoke a linear entropy penalty.
arXiv:2609. 12785v1 Announce Type: new Abstract: Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice.
By Misbah Uz Zaman, Anirbit Mukherjee
arXiv:2603. 06009v2 Announce Type: replace Abstract: An agent's performance stagnating at a suboptimal level is a common problem in deep on-policy RL.
By Michael Beukman, Khimya Khetarpal, Zeyu Zheng, Will Dabney, Jakob Foerster, Michael Dennis, Clare Lyle
arXiv:2607. 09097v1 Announce Type: cross Abstract: We study stochastic fixed-point equations $\mathbf{T}(\mathbf{x}) = \mathbf{x}$ over normed spaces $(\mathcal{E}, \|\cdot\|)$, where the operator $\mathbf{T}$ is nonexpansive or contractive and is accessed only through unbiased stochastic evaluations with bounded second central moment.
By Jelena Diakonikolas
arXiv:2409. 19279v2 Announce Type: replace-cross Abstract: Continuous-time models can reveal accelerated structures in distributed optimization, but their rates need not survive direct discretization.
By Kushal Chakrabarti, Mayank Baranwal