arXiv AI
MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 11
Toward a First-Principles Update Geometry for the Language-Model Head
The paper proposes a new update geometry for the language‑model head by treating the head and softmax as a single module and using Hilbert’s projective distance to measure functional change. It replaces the spectral norm with the Euclidean row diameter, derives a tractable RowNorm update rule, and demonstrates that RowNorm substantially reduces step diameters and Hilbert perturbations while only slightly increasing validation loss.
arXiv Machine Learning
Oct 3
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
AF‑Muon is an AdamW‑free extension of the Muon optimizer that retains Muon’s matrix update for hidden weights while applying a support‑aware finite‑cap linear minimization oracle to tied vocabulary tables and an RMS‑normalized update for one‑dimensional auxiliary parameters. This design eliminates second‑moment state, reducing optimizer‑state memory by about 20% compared to Hybrid Muon. Across nine tied‑token settings—including decoder‑only language models, T5‑style encoder‑decoders, and ImageGPT‑style variants—AF‑Muon consistently improves mean validation loss and perplexity over both Hybrid Muon and a SCION‑style Sign endpoint, with robust gains confirmed by long‑horizon runs and hyperparameter studies.
arXiv AI
Jul 23
An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies
arXiv:2607. 19771v1 Announce Type: cross Abstract: Muon and related matrix-sign optimizers are increasingly used to pre-train large language models, but their effect on the internal geometry of individual weight matrices is not well understood.
arXiv AI
Oct 7
Muon Is Theoretically Wrong For Convolutions, But Empirically Effective
arXiv:2610.07103v1 Announce Type: cross Abstract: Muon, an optimizer known for its efficiency, has a clear interpretation for matrix-valued updates, but convolutional kernels are stored as four-dimen...
arXiv Machine Learning
Jun 30
Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
arXiv:2606. 30509v1 Announce Type: new Abstract: Matrix factorization (i.
arXiv Machine Learning
Jun 25
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.