← Back to all news
arXiv Machine Learning By Aryan Das, Vinay Kumar Verma, Aakansha Mishra

MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

Read the original on arXiv Machine Learning →

The paper introduces Muon-RT, a variant of the Muon self‑distillation method that eliminates the need for a separate momentum buffer by reusing the teacher‑student displacement. By removing the radial component, normalizing spectrally, and mixing with the current gradient after Newton‑Schulz orthogonalization, Muon‑RT achieves linear contraction of the objective gap under certain assumptions. Experiments on Qwen2.5‑7B‑Instruct and Olmo‑3‑7B‑Instruct show that Muon‑RT outperforms AdamW and momentum Muon while reducing persistent memory usage by up to 42%.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 25

A Physical Response-and-Memory Model for Muon Optimization

The paper introduces a physical response-and-memory model for the Muon optimizer, explaining its semi‑orthogonalized momentum update as the maximally dissipative direction under an output‑side safety budget. It treats the weight matrix as a responsive medium with internal stress, showing that momentum corresponds to accumulated stress whose relaxation occurs over multiple timescales—fast and slow. Based on this, the authors propose the Bi‑Maxwell optimizer, which uses a two‑timescale memory kernel and achieves target loss in fewer steps on a public large‑language‑model benchmark.

By Yinze Hu, Hongjun Xiang, Xingao Gong, Hongyu Yu
llmsbenchmarkssafety
More like this →
arXiv Machine Learning
Sep 2

Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining

The paper introduces two variance‑adaptive variants of the Muon optimizer—Muon‑NSR and Muon‑VS—for language model pretraining. Both methods incorporate gradient‑variance information into Muon’s orthogonalization process without adding extra hyperparameters, preserving its spectral normalization structure. Experiments on Llama‑style and GPT‑2 models ranging from 125 M to 1.2 B parameters show that these variants outperform well‑tuned Muon baselines and achieve up to a 1.33× step‑to‑target speedup on Llama‑1.2B.

By Jingru Li, Yibo Fan, Huan Li
llms
More like this →