arXiv:2601. 12238v5 Announce Type: replace-cross Abstract: In this paper, we provide a comprehensive theoretical analysis of Stochastic Gradient Descent (SGD) and its momentum variants (Polyak Heavy-Ball and Nesterov) for tracking time-varying optima under strong convexity and smoothness.
By Sharan Sahu, Cameron J. Hogan, Martin T. Wells
arXiv:2606. 10085v1 Announce Type: new Abstract: Matrix-valued time series arise in a wide range of applications, such as spatio-temporal data from medical imaging and geophysics.
By Zhen Qin, Yang Chen
arXiv:2606. 28879v1 Announce Type: new Abstract: The adaptive moment estimation algorithm, known as Adam, is widely used in modern machine learning, owing to its low per-iteration complexity and strong empirical performance.
By Xin Zheng, Yifei Jin, Lei Guo
The paper studies a two‑stage learning framework that first trains an offline model using approximate nonlinear‑least‑squares estimation and then adapts it online with a meta‑LMS algorithm to handle parameter drift in nonlinear stochastic dynamical systems. It provides an upper bound on the offline generalization error that accounts for strong data correlation and distribution shift via Kullback‑Leibler divergence, and it demonstrates that the combined offline‑online approach outperforms methods that rely solely on offline or online learning. Both theoretical analysis and empirical experiments support the claimed performance gains.
By Haizheng Li, Lei Guo
arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.
By Ganzhao Yuan
The paper presents a theoretical study of Adam in non‑stationary stochastic optimization, distinguishing two regimes: Euclidean tracking under adaptive strong monotonicity and high‑probability projected stationarity for general smooth objectives. It derives finite‑time bounds that decompose into initialization, objective drift, first‑moment tracking error (β₁), and preconditioner perturbation (β₂), and characterizes burn‑in times for constant and step‑decay schedules. The analysis reveals a noise–drift tradeoff, showing that in noise‑dominated settings Adam’s adaptive mechanisms can improve guarantees, while in drift‑dominated settings they may worsen tracking, potentially making vanilla SGD preferable.
By Sharan Sahu, Abir Sarkar, Cameron J. Hogan, Martin T. Wells