Hugging Face Trending Papers
Optimizing Large Language Models with Chained LMOs
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv AI
5d ago
Optimizing Large Language Models with Chained LMOs
The paper introduces chained linear minimization oracles (chained LMOs) as a unified framework for optimizers that compose multiple matrix normalizations, addressing fragmentation in the field. It shows that while many such chains can diverge on smooth convex objectives, chaining can still improve performance under anisotropic embeddings, as demonstrated by the new TensorChain optimizer. TensorChain, which stacks compatible weight matrices across layers and normalizes a 3‑D tensor, achieves a 9.6% token efficiency gain over Muon in Qwen3 0.6B and 1.7B pretraining at matched validation loss.
arXiv Machine Learning
Sep 23
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
MONA is a new optimizer that extends the Muon optimizer by adding a Nesterov‑style acceleration term derived from an exponential moving average of gradient differences. The paper provides a convergence analysis showing that this term offers curvature‑aware corrections while maintaining Muon’s spectral‑norm regularization. Empirical results demonstrate that MONA outperforms both Muon and AdamW on Mixture‑of‑Experts pretraining across models ranging from 1 B to 68 B parameters, and achieves state‑of‑the‑art performance on downstream benchmarks after fine‑tuning the largest model.
arXiv Machine Learning
Jun 25
Tensorion: A Tensor-Aware Generalization of the Muon Optimizer
arXiv:2606. 25975v1 Announce Type: new Abstract: Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models.
Hugging Face Trending Papers
Jun 24
Tensorion: A Tensor-Aware Generalization of the Muon Optimizer
Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models. Recent work has shown that exploiting matrix structure can improve optimization dynamics.
arXiv AI
2d ago
From SGD to Muon: Adaptive Optimization via Schatten-p Norms
The paper introduces a data‑driven criterion that dynamically selects optimal Linear Minimization Oracle (LMO) geometries for each layer of a deep neural network, bridging the gap between SGD and Muon updates. By deriving the criterion from gradient and activation statistics via a single‑step random feature regression surrogate, the authors enable a design space that interpolates between existing optimizers, including SGD, Muon, Adam, and MuAdam. The adaptive approach incurs only about a 3% runtime overhead and demonstrates competitive or superior performance compared to the best static optimizer across three training scenarios.
arXiv Machine Learning
Jun 26
DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
arXiv:2606. 27153v1 Announce Type: cross Abstract: Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads.