arXiv AI
Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Oct 13
Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry
Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearity that follows. This...
arXiv Machine Learning
Sep 11
Toward a First-Principles Update Geometry for the Language-Model Head
The paper proposes a new update geometry for the language‑model head by treating the head and softmax as a single module and using Hilbert’s projective distance to measure functional change. It replaces the spectral norm with the Euclidean row diameter, derives a tractable RowNorm update rule, and demonstrates that RowNorm substantially reduces step diameters and Hilbert perturbations while only slightly increasing validation loss.
arXiv Machine Learning
Aug 10
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
arXiv:2507. 07814v2 Announce Type: replace Abstract: We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions.
arXiv AI
Oct 5
MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads
arXiv:2610.02705v1 Announce Type: cross Abstract: The Muon optimizer derives its update rule for hidden linear layers by solving a local linearization of the loss penalized by the spectral norm, moti...
arXiv Machine Learning
Jun 9
Convergence Bound and Critical Batch Size of Muon Optimizer
arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.
arXiv Machine Learning
Aug 11
Training-Free Universal Approximation by Prompting Random Transformers
arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?