The paper introduces RanSOM, a second‑order momentum algorithm that eliminates the need for a Lipschitz Hessian or auxiliary forward/backward passes by randomizing the step length. By sampling the step from an exponential distribution and using a Stein‑type identity, the method obtains an unbiased estimate of the gradient change with a single Hessian‑vector product that also serves the next gradient computation. RanSOM-E achieves optimal oracle‑complexity exponents for heavy‑tailed noise without clipping, while RanSOM-B preserves feasibility for Frank‑Wolfe iterates, and experiments on CIFAR‑10 show comparable accuracy to existing methods with lower update time.
By El Mahdi Chayti, Mohamad Fakhouri
arXiv:2610.33477v1 Announce Type: cross
Abstract: We present a generic actor-critic method that combines a ridge-regularized pathwise least-squares temporal-difference (LSTD) critic with importance-c...
By Yanjie Dong, Zhou Su, Wei Wang, Gang Wang
The paper analyzes the stochastic optimizer Muon, which orthogonalizes an exponential moving average of minibatch gradients, and compares its behavior to the baseline SignSGD on a high‑dimensional matrix‑valued least squares problem. It derives deterministic dynamics showing Muon’s sensitivity to an effective batch size and its failure to resolve small eigenmodes when whitening is exact and the effective batch is small, while Momentum and finite Newton–Shulz iterations can mitigate this. The study presents a phase diagram in terms of data and target exponents, indicating when Muon or SignSGD is preferable based on covariance spectrum steepness and target loss scaling.
By Elliot Paquette, Noah Marshall, Lucas Benigni, Guangyuan Wang, Atish Agarwala, Courtney Paquette
arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.
By Ganzhao Yuan
arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.
By Naoki Sato, Hiroki Naganuma, Hideaki Iiduka
arXiv:2609.39595v1 Announce Type: new
Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...
By Hanyng Peng, Hui Wang, Yue Yu