The paper introduces a unified framework for first‑order optimization algorithms applied to nonconvex unconstrained problems. It incorporates adaptively preconditioned gradients and covers popular methods such as full and diagonal AdaGrad, AdaNorm, and an adaptive variant of Muon. The framework supports heterogeneous geometries across variable groups and provides a fully stochastic global convergence analysis for all methods, with or without two types of momentum, under reasonable variance assumptions without requiring bounded stochastic gradients or small step sizes.
arXiv:2607. 08104v1 Announce Type: new Abstract: Stochastic gradient descent (SGD) is a cornerstone of modern optimization.
arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.
arXiv:2406. 13041v3 Announce Type: replace Abstract: Lower-bound analyses for nonconvex strongly-concave minimax optimization problems have shown that stochastic first-order algorithms require at least $\mathcal{O}(\varepsilon^{-4})$ sample complexity to find an $\varepsilon$-stationary point.
A new class of asynchronous adaptive first-order optimization methods is introduced, comprising asynchronous variants of several popular algorithms. Versions of these methods using momentum and/or inexact normalization are also considered.
arXiv:2602. 10204v2 Announce Type: replace Abstract: We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization.
arXiv:2606. 01787v1 Announce Type: new Abstract: A new class of asynchronous adaptive first-order optimization methods is introduced, comprising asynchronous variants of several popular algorithms.
arXiv:2509. 14969v2 Announce Type: replace Abstract: We introduce a new adaptive step-size strategy for convex optimization with stochastic gradient that exploits the local geometry of the objective function only by means of a first-order stochastic oracle and without any hyper-parameter tuning.
arXiv:2606. 15702v1 Announce Type: cross Abstract: Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm design and theoretical analysis.
arXiv:2605. 18694v2 Announce Type: replace-cross Abstract: Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process.
arXiv:2604. 08742v2 Announce Type: replace-cross Abstract: Adam is widely used, but its convergence theory remains incomplete even in the deterministic full-batch setting because momentum and adaptive preconditioning are tightly coupled.
arXiv:2609.08277v1 Announce Type: new Abstract: We study zeroth-order optimization of non-convex functions with the aid of directional hints, which are cheap but potentially inaccurate approximations...
arXiv:2609.39595v1 Announce Type: new Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...