The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
arXiv:2610.17588v1 Announce Type: cross Abstract: Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full mo...
The paper proposes a new update geometry for the language‑model head by treating the head and softmax as a single module and using Hilbert’s projective distance to measure functional change. It replaces the spectral norm with the Euclidean row diameter, derives a tractable RowNorm update rule, and demonstrates that RowNorm substantially reduces step diameters and Hilbert perturbations while only slightly increasing validation loss.
arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.
arXiv:2610.02705v1 Announce Type: cross Abstract: The Muon optimizer derives its update rule for hidden linear layers by solving a local linearization of the loss penalized by the spectral norm, moti...
The paper introduces a data‑driven criterion that dynamically selects optimal Linear Minimization Oracle (LMO) geometries for each layer of a deep neural network, bridging the gap between SGD and Muon updates. By deriving the criterion from gradient and activation statistics via a single‑step random feature regression surrogate, the authors enable a design space that interpolates between existing optimizers, including SGD, Muon, Adam, and MuAdam. The adaptive approach incurs only about a 3% runtime overhead and demonstrates competitive or superior performance compared to the best static optimizer across three training scenarios.
arXiv:2507. 07814v2 Announce Type: replace Abstract: We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions.