arXiv:2608. 12710v1 Announce Type: new Abstract: Muon, a more recently developed optimizer, is useful for matrix-wise models in AI areas.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
arXiv:2509. 14562v4 Announce Type: replace Abstract: Large models recently are widely applied in machine learning, so efficient training of large models has received widespread attention.
arXiv:2608. 04607v1 Announce Type: cross Abstract: Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs).
FedSubMuon introduces a communication‑efficient federated fine‑tuning approach for large language models by optimizing compact coefficient matrices within shared structured subspaces, thereby keeping Muon’s matrix‑aware optimization while reducing client upload size. An accuracy‑oriented variant, FedSubMuon‑GT, further adapts subspace bases using projected gradients to better align with task‑relevant directions. Experiments on instruction tuning and mathematical reasoning demonstrate that FedSubMuon‑GT achieves the best overall accuracy on most dataset‑model pairs, while FedSubMuon outperforms all matched‑budget baselines and reduces communication by up to 5.5× on Llama‑1B and 1.4× on Qwen‑4B compared to the closest baseline.
arXiv:2606. 27153v1 Announce Type: cross Abstract: Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads.
arXiv:2606. 25975v1 Announce Type: new Abstract: Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models.
The paper introduces low‑rank orthogonalization, a technique that exploits the low‑rank nature of gradients in neural network training to perform matrix orthogonalization more efficiently. Building on this, the authors present low‑rank matrix‑signed gradient descent (MSGD) and a low‑rank variant of the Muon optimizer, showing through experiments that low‑rank Muon matches or surpasses vanilla Muon on GPT‑2 and LLaMA pretraining, especially for larger models. Theoretical analysis provides iteration‑complexity bounds for both low‑rank MSGD and low‑rank Muon under heavy‑tailed noise.